1829 Commits
Author SHA1 Message Date
discord9 25263af9d0 feat(promql): fold and average native histograms in the native aggregate (#10199)
* feat(promql): carry a native histogram in aggregate state

`sum` and `avg` over native histograms need a partial state whose payload is
variable width, which the numeric state family cannot express. Add two kinds,
`histogram_sum` and `histogram_avg`, whose components are a live histogram, a
layout-carrying compensation, and the evidence a merge records.

A position's payload is one `StatePayload`, either the numeric components a
float kind carries or a boxed histogram state, so absence stays a single
`None` and a field never holds both families. The numeric encoding is byte for
byte what it was, and a histogram component nests the same struct a histogram
sample frame already uses.

The compensation must carry the live value's bucket layout: the kernel indexes
it by the value's positions, so a placeholder without bucket indices would be
an out-of-bounds read rather than a loss of precision. Both the producer and
the decoder check the schema, the zero threshold, the custom bounds and the
expanded logical bucket indices, which accepts two spellings of the same
buckets and refuses two different layouts.

State accounting charges the payload's real width instead of a fixed slot, so
a wide bucket layout is paid for. `HistogramSum` cannot declare
`incremental_mean`, which is checked structurally: a sum has no mean mode.

No admission, split or runtime wiring yet; nothing produces these states until
the aggregate kernels land.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): fold native histograms into a `sum` aggregate

`sum` over a native histogram now folds in the native aggregate: the first
sample at a grid position establishes the accumulator from a layout-carrying
zero compensation, and every later one is added with a compensated add whose
receiver is the live value.

The evidence the final answer needs is recorded as it appears: set when a merge
reconciled custom bucket bounds, and both reset-hint bits when samples reported
a reset and reported that none happened. Two layouts that cannot be added -
exponential and custom buckets in one position - poison the point, which the
final answer drops instead of answering a partial sum.

A numeric overflow is not a fault and does not poison anything: `sum` answers
`Inf` when its cumulative sum saturates, exactly as the kernel's own sum does.

`avg` keeps refusing a histogram: its mean mode needs the merge rules, which
are not part of this change. A group that kept histogram payloads now refuses
to export a partial numeric state rather than exporting an empty one.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): average native histograms in the native aggregate

`avg` over a native histogram folds the same way the v3.15.0 kernel averages a
range: sum with compensation while the sum fits, then switch to a running mean
once the next add would overflow.

The switch keeps the answer finite where the sum would not be: two histograms
whose scalar sum is `1e308` each average to `1e308`, not to `Inf`. It starts the
mean from the sum accumulated so far divided by the samples that produced it,
and weights every later sample by the fraction of samples it represents, so the
compensation is scaled by the same weight as the value it belongs to.

The complete answer divides both the sum and its compensation by the sample
count, or answers the mean with its compensation. `start_timestamp` is cleared
and the payload compacted, as every other derived histogram is.

No operation refuses a histogram sample any more, so the refusal variant is
gone rather than left unused: what a fold accepts is decided by its arithmetic,
not by the sample's type.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): merge native histogram state across a split

A split aggregate exports its partial state and folds it somewhere else, so a
histogram fold has to travel. The binding now says which layout a node carries:
`aggregate_state_binding(kind, field_kind)` picks the histogram family for a
histogram input and the numeric family otherwise, and `BoundAggregate` records
the kind its direct input answers with, decided where the node is bound.

The partial half exports the live histogram, the compensation that carries its
bucket layout, and the evidence the merge needs; a sum cannot claim a mean mode
because its kind has no such component. The final half merges two partials the
way the v3.15.0 kernel merges contributions: two compensated adds for a sum, and
a mean path that converts each side, turns its count into a weight, and scales
before adding so that a pair of `1e308` histograms still averages to `1e308`. A
sum/sum pair first runs the same trial the complete fold runs, and only keeps it
when it does not overflow.

An unreconcilable layout poisons the point in either partial: the merge keeps it
poisoned and never revives it, and the answer drops that point. Evidence is
merged bit by bit, including from a poisoned side, so which annotation fires
does not depend on the merge order.

The node that finishes the answer is the only one that reports: `Complete` and
`Final` speak after their output is queued, `Partial` stays silent because its
evidence travels in the state. A poisoned point suppresses its reconciliation
info, which is what makes the message set order-independent.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): admit a native `sum` and `avg` over histograms

A payload-folding aggregate is now admitted exactly when it binds histogram
state: `sum` and `avg` over a native histogram fold and merge their own state,
so the gate that refused every such input now asks what the node will build
rather than what its input carries. Their answer column is the canonical
histogram struct; `count`, `group`, `min`, `max`, `stddev` and `stdvar` keep
answering floats whatever they read.

The split may carry a histogram answer when the cut aggregate exports histogram
state. That is also what makes the state schema's value column count follow the
kind, which a histogram split otherwise failed against.

A histogram fold the split declines is not segmented. Segmentation answers a
group by folding each segment and appending the pieces, so a point whose
contributions are spread over partitions would be folded once per segment - a
sum would merge histograms that never met, an average would average averages.
The whole statement therefore keeps the plan it had, which also covers an
aggregate above an aggregate and a whole-vector ranking above a fold: neither
can cross the split boundary, and cutting them would answer a piece no segment
owns.

Multi-input histograms and a histogram behind a canonical merge keep their
refusals: the state crossing a canonical boundary has no execution coverage.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): read a native histogram sum and average over HTTP

Three cases write native histograms through Remote Write 2.0 and read the
answer back through the Prometheus query API. Each one asserts the native route
really executed: the observer records the split's two fused producers, and the
one publishing a histogram column is the final consumer, so a statement that
fell back to the existing plan cannot satisfy the test.

`sum` and `avg` answer the values the v3.15.0 kernels produce, and both are
compared against the same query without the native hint. That comparison uses a
relative tolerance for `avg` and an exact one for `sum`, because the existing
plan averages by weighting each sample (`12 * (1/3)` is one ulp below `12 / 3`)
while the native fold divides at the end - the difference this change documents
rather than hides.

The partitioned case flushes after every write so the scan really splits, then
asserts the partial producer reported output in more than one partition and that
the same query at parallelism one answers the same JSON.

Two layouts in one metric remove the group and report it: the answer is empty
and the response's warnings name the incompatible layouts, asserted after the
native nodes are known to have run.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): decide a reset collision per point and pin the avg merge weights

The contradictory-reset warning was decided from the two evidence bits OR-ed across every
group and every point, so sum by (job) (h) with CounterReset only in job a and
NotCounterReset only in job b warned, as did one series crossing its reset over a range.
Prometheus checks counterResetSeen && notCounterResetSeen per group per step, so the bit is
now decided per accumulator and only the decision is OR-ed across the answer. The state
layout is untouched: it still carries the two raw bits, and the collision is settled when the
answer is.

Nothing caught a swapped pair of avg merge weights: the existing overflowing-pair test gives
both partitions one sample and the same mean, so both weightings answer 1e308. The new case
merges a two-sample partition that switched to a mean with a six-sample one that still sums,
weights 2/8 and 6/8, and asserts 7 * 2^1019; swapping the weights answers 13 * 2^1019 and
fails.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): merge histogram partials through the accumulator, not a row encoder

The eight histogram merge tests drove their partials through a test-side encoder that writes
numeric state rows, so they panicked the moment a state carried a histogram payload. That
encoder never produced a histogram row for anyone: the merge they want to check is the one
HistogramAccumulator::merge_state performs, and the row path is the executor's.

They now merge the payloads directly -- from_payload for the first, merge_state for the rest
-- and read the answer through finished_histogram and the evidence through the production
summary. Every assertion keeps its value and strength: the unequal-count case still asserts
7 * 2^1019, so swapping the merge weights answers 13 * 2^1019 and fails, and the evidence
case still fails when the merge order decides the result.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): drive histogram state rows through merge_row

The cross-partition merge reads each partition's state frame as a StateRow, and
merge_histogram_row is the code that decodes a histogram payload out of one. No test reached
it: the numeric row encoder panics on a histogram, and the histogram merge cases go through
the accumulator instead, so a bug in that decode -- a swapped component, a poison bit lost, a
field index off by one -- would have left every test green.

These construct StateFieldsBatch rows directly and merge them. Sum and the average's mean
path assert the same bits the accumulator answers for the same payloads, so the two paths
cannot drift; a poisoned row is asserted to poison the point; a row missing a component and a
row whose compensation carries another bucket layout are asserted to be refused rather than
answer something. Dropping the poison bit, not decoding the mean mode, and reversing the
field index each fail one of them.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): bind an aggregate's state from the statement's own field kind

The rebase replayed the compiler call with a literal Some(FieldKind::Float) for the
statement's source kind, so every aggregate bound float state: sum over a native-histogram
column asked the gate whether it folds a payload, answered yes, and the statement fell back
to the legacy plan with prom_native_histogram_agg_sum in it.

The call now passes statement_kind, the kind the prepared scans report for the statement's
fields. The duplicate guard on the segmentation path -- which refused every histogram fold
rather than only the ones the split declined -- is removed, and the mutation scripts the
earlier commits picked up are dropped.

Signed-off-by: discord9 <discord9@163.com>

* test(http): register the three native histogram aggregate cases

Splicing the range test and this branch's cases together during the rebase kept the function
bodies but dropped their http_tests! entries, so the three cases existed without ever running.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): identify the histogram column by type, not by name

Native result columns are named after their expressions now, so the three admission tests
that looked for a column called greptime_native_histogram found none in a plan that answered
exactly the histogram they meant to check. They look for the histogram value type instead,
which is what the assertions were about.

Signed-off-by: discord9 <discord9@163.com>

* test(http): find the histogram answer by type and allow an empty scan partition

`answer_consumer` looked for a column named `greptime_native_histogram`, which the answer is no
longer called now that result columns take their expression's name (#10231). It reads the histogram
value type instead, and only the schema's own fields: a state frame keeps its histogram inside
`fields`, so a top-level type check still finds the consumer and not the producer.

The partition case also required every scan partition to hold rows. Routing hashes each series'
label set, so eight series over four partitions can leave one empty; #10469 stopped asking for
non-empty partitions elsewhere and `test_remote_write_v2_histogram_native_range_function` already
reads the same fixture this way. The producer still has to run on more than one partition, and the
answer still has to count every series once.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): carry the histogram fold onto #10433's bound aggregate

Rebasing onto #10433 left three things behind. `BoundAggregate` gained
`field_kind`, which the quantile tests construct without; the value is `Float`
because those fixtures hand the fold float samples, and the one call to
`aggregate_state_binding` now names the kind it asks about.

`native_composed_binary_admits_complex_operands` also expected
`hist * on(tag_0) sum by (tag_0) (hist)` to fall back, for a reason this branch
made false: it said `sum` has no histogram state to fold into. It has one now,
so the pair composes. `histogram * histogram` is not an error in Prometheus -
only `+` and `-` apply and any other operator removes the element with an info
annotation - so the composed pair is admitted and keeps the legacy float layout
for the answer no sample can fill. The test asserts that shape instead, and the
refusal it still checks (`sum(sum(hist))`) is named for what refuses it: the
split cannot carry an aggregate above a split aggregate.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): carry the histogram fold onto #10202's plan-line display

`BoundAggregate.field_kind` again: the display test's `sum(rate(...))` builds one without
the field, and its input is a histogram, so the field says so.

`native_fused_quantile_is_never_folded_per_partition` looked for the statement's text in the
display to show the fold is still named by the quantile it wrote. #10202 renders the tree as plan
lines instead, so the quantile is named by its own line - `Aggregate: quantile 0.5` - and that is
what the assertion reads now.

Signed-off-by: discord9 <discord9@163.com>

* style(http): keep the answer-column doc off a list item

The first line of the histogram sentence ended on a hyphen, so the next line began with one and
clippy read it as an unindented list item.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 18:29:58 +08:00
discord9 bd1ad73682 perf(promql): cut per-series fixed costs in decoding, packing and accounting (#10416)
Signed-off-by: discord9 <discord9@163.com>
2026-11-17 17:44:02 +08:00
discord9 7ba85f6369 feat(promql): show the native tree as plan lines with per-node metrics (#10202)
feat(promql): draw the native tree as plan lines with per-node metrics

The fused node prints its own compiled tree instead of echoing the PromQL text it
was built from: one node per line inside a `native_tree=[ ... ]` block, with each
node's figures on its own line when the statement ran. A split aggregate's two
halves therefore show the trees they compute rather than the same statement, and
EXPLAIN/ANALYZE of a native statement reads like the plan it runs.

The statement text stays on the node as data: it is what names the value columns
(`expression_field_names`), so it has to survive to execution and over the wire
where a node is rebuilt from its envelope. What changes is that it is no longer
what the node prints.

The tree's registrations sit in the node's own metric set, so a reader that keys
by name reads that set; `ExecutionPlan::metrics` leaves them out because a flat
set cannot say which node recorded what.

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 16:28:01 +08:00
discord9 089121c613 Merge main into narthex
Conflicts:
- `src/common/query/Cargo.toml`: `futures-util` stays non-optional because the
  series-frame API exposes `futures_util::Stream`; `testing` only pulls
  `datafusion-datasource`.
- `src/frontend/src/instance.rs`: import union (`ServerMemoryLimiter` from main,
  `MetricDiscoveryScope` from narthex).
- `Cargo.toml` / `Cargo.lock`: follow main's newer greptime-proto rev
  (`c46f7ee9c`), which contains the `RawSeriesSource` this branch needs.
- `tests-integration/src/standalone.rs`: keep both `query_options` and
  `auto_create_physical_table_timestamp_unit`.

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 10:13:41 +08:00
fys 453c35e9d3 feat: support trigger cache invalidation through frontend plugins (#10460)
Signed-off-by: fys <fengys1996@gmail.com>
2026-11-16 23:14:07 +00:00
jeremyhi c6962802a9 feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration

A region migration now gives its candidate region registration a random non-zero
candidate lease id. Leases granted because of that registration carry it, but only
while the table route exists, and the upgrade instruction carries the same id. The
datanode passes the id of each lease to the engine, and Mito records the lease of
the latest heartbeat per region outside the region instance, so it survives a
reopen during catchup.

The migration also drops the candidate registration before it restores the route
on rollback, and before the procedure leaves the migration tracker when it ends
without a rollback.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a lease only while the region stays open

Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* build: point greptime-proto at the candidate lease id change

Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop lease records when workers stop

Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): drop a redundant helper comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(store-api): pin the candidate lease id wire bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: document candidate registration and lease record lifecycles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep candidate lease ids through customized lease renewers

The heartbeat handler now sets the candidate lease id of each lease a
customized renewer returns from the renewed region with the same id.

The metric engine records a lease for the data region only when the
metadata region ignored it, since the data region records its own
leases. The countdown reset command carries a named struct instead of
a tuple.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): cover a lease the data region ignores

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tighten lease id lookups and lease test assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: align lease naming and share the txn-observing kv backend

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tidy lease naming, docs and shared test helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop field comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop method comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: say which heartbeat a lease's send time refers to

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop the reset deadline field comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: state when an operating region's role and id reach a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: pin greptime-proto to the merged candidate lease id change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: add the candidate lease id argument to the direct demotion test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: order new keeper helpers after public items and test equal send times

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 18:37:17 +00:00
Weny Xu 610794e447 perf: batch logical table cleanup when dropping databases (#10461)
* perf: batch logical table cleanup when dropping databases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refine drop database batching and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify drop database traversal and batch sizing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: tighten drop database batch size estimate

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: narrow logical table metadata cleanup API

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 14:17:35 +00:00
shuiyisong 0a93810c23 fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): reserve memory for table batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): prefix the labels column

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): hide exemplar table from metadata

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): reconcile source tags before writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): prune tags and retain source IDs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): record slow queries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: account for held memory in batch admission

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: config comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: simplify exemplar and memory tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: deduplicate exemplar setup

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-16 11:55:20 +00:00
discord9 2e4141e77d refactor(promql): move the segmentation engine into the promql crate (#10418)
Lands the segmentation-engine move together with the uniform native-PromQL topology it builds on (#10321), which this branch carries as its base.
2026-11-16 17:36:28 +08:00
Weny Xu 89e14ab936 fix(mito2): cancel compaction merges at batch boundaries (#10411)
* fix(mito2): cancel compaction merges at batch boundaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve upload errors and expose merge cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): normalize encoder cancellation at SST writer boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(mito2): use imports for SST write types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): share cooperative upload cancellation with tokens

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): classify nested upload cancellation errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 08:32:11 +00:00
discord9 5517c4b76f refactor(promql): let the legacy histogram path read the native kernel (#10420)
* refactor(promql): read the legacy histogram estimators from the native kernel

`prom_native_histogram_quantile` and `prom_native_histogram_fraction` estimated from
`common_query`'s `NativeHistogram` bucket walk: exponential bucket bounds computed as `2^x`, and a
forward walk from the lowest bucket only. The native kernel in `histogram_eval` walks the same
histogram the way Prometheus does -- the bound table in `bounds`, and a walk from the highest bucket
down once a finite-sum quantile reaches the median -- so one histogram had two answers that differ
in their last places.

Read the `NativeHistogram` through the borrowed `HistogramView` both representations share and
return the kernel's answer, mapping its outcome back onto the NaN-observation notes the legacy UDFs
record. Their signatures and the notes are unchanged; only the arithmetic behind them moves.

The `2^x` estimator in `common_query::native_histogram` stays for that crate's public API and is
annotated as such: no PromQL path reaches it any more.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): delegate the legacy histogram arithmetic to the native kernel

`prom_native_histogram_add`, `_sub`, `_mul_scalar`, `_scalar_mul`, `_div_scalar`, `_neg`, the
sum/avg accumulators, `delta`/`idelta`/`rate`/`increase`, and the weighted mean that merges partial
states all scaled and combined `NativeHistogram`s with that type's own reconciliation: its own
zero-bucket widening, schema reduction and custom-bound intersection, over `2^x` bounds.
`FloatHistogram::add`/`sub`/`mul`/`div` already implement Prometheus's rules (`addBuckets`,
`reconcileZeroBuckets`, `CopyToSchema`, `intersectCustomBucketBounds`) and the native kernel's
golden tests cover them, so this keeps one implementation of the semantics.

`AddOutcome` folds into the existing semantics rather than being dropped: `custom_bounds_reconciled`
is the intersection the kernel keeps in `custom_values`, which the callers already announce from the
operands, and `counter_reset_collision` is the contradiction the kernel answers with the unknown
hint the legacy arithmetic derived from two hints that disagree; both are asserted in debug builds.
The public `NativeHistogram` signatures are unchanged, the sums are compacted as before so
`promql_eq` and the serialized spans keep their answers, and unary minus goes through the kernel's
multiplication like the other scalings.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the legacy delegation's observable answers

Comparing the bridge against the kernel it calls only shows they agree today; it cannot show
what either answer means. These tests pin the answers themselves.

- quantile with a NaN sum over several buckets answers 32.0 with NaNSkew, because the NaN walk
  leaves the last bucket for the interpolation; below the rank it answers NaN with NaNResult.
- an exact tie answers 1.0 through the legacy walk and 2.0 through the kernel, on a fixture with
  an empty bucket between the two, and neither records a note.
- assert_same_payload now compares every field, including the start timestamp policy, custom
  values, the complete span segmentation and the bucket bits.
- round-tripping preserves reset hints, custom values, zero-length spans and gaps.
- add/sub reconcile the reset hint and the custom bounds, compact cancelled buckets, and keep
  promql_eq's expanded-index equality.
- scale/divide_by keep the field-by-field fallback for an unreadable layout -- mismatched counts,
  a backwards span, an overflowing index, a negative length -- and preserve its start timestamp;
  add/sub answer None for the same layouts. Zero, negative and NaN factors are pinned.
- rate/increase/delta/idelta/irate clear the start timestamp of a derived histogram, while a
  single-sample sum aggregate keeps its stored one.

A layout whose later span steps backwards is refused before the kernel sees it: the legacy
bucket_indices rejected it, while side_from_view would have accepted repeated indices.

Signed-off-by: discord9 <discord9@163.com>

* test(http): pin the legacy histogram route's rate, NaN quantile and derived windows

The delegation changes what three kinds of well-formed data answer, so assert it where a user
sees it. Every request runs without the native hint and asserts !executes_fused_series on the
observed plan, which proves the legacy route rather than assuming it from the missing hint.

- rate and increase over a raw sample with a start timestamp inside the window still extrapolate
  from the synthetic zero: count 8/3 and sum 10/3 for a rate, 8 and 10 for an increase, with no
  infos or warnings.
- histogram_quantile(0.5) over schema 0, buckets [8, 1], count 10 and a NaN sum answers 32.0 and
  reports the NaN-observation info, matching Prometheus's walk that leaves the last bucket for
  the interpolation. The previous estimator answered 2^-0.375.
- rate over a raw window answers one series while the same window over h*1 answers none: the
  derived sample carries no start timestamp, so it has no synthetic zero to extrapolate from.
  Both phases are covered, memtable and flushed SST.

The native-histogram test now installs the route interceptor the way its sibling tests do, so the
plan can be observed; all of its original assertions are unchanged.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 16:26:52 +08:00
Jingbo ChaiandClaude Opus 5.5 7e177b06fe fix(telemetry): avoid spurious log directory error on first start (#10446)
* fix(telemetry): avoid spurious log directory error on first start

Since 0.2.4, tracing-appender prunes old log files when an appender
with max_log_files is built, and it lists the log directory before
creating it. When the directory does not exist yet, it prints "Error
reading the log directory/files" to stderr. GreptimeDB sets
max_log_files by default, so every start with a missing log directory
printed this line.

Create the log directory in build_file_appender before building the
appender. Creation errors are left to the appender build, which
reports them as before. Add a test that checks the stderr of a child
process.

Works around https://github.com/tokio-rs/tracing/issues/3612.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(telemetry): drop the first-start log directory test

Review feedback: the test is not needed for such a minor change.
Restore the unix-only test imports that it had merged.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 01:38:28 +00:00
dayan02devanddennis zhuang dbd080b840 fix(recordbatch): treat -0.0 and +0.0 as equal in SimpleFilterEvaluator (#10443)
* test(recordbatch): compare float zero predicates with DataFusion

`SimpleFilterEvaluator` filters SST rows exactly, so it has to agree
with the SQL predicate that DataFusion evaluates above the scan.
DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons
against both zeros, NaN and NULL in Float64 and Float32 columns, and
check the selected rows against DataFusion's filter.

On main this fails: `v = Float64(0)` selects only the +0.0 row, while
DataFusion selects both zeros.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters

DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison
kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator
called the kernels directly, and for floats they compare bit patterns:
-0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as
an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)`
dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding
+0.0. Rows still in a memtable matched.

Normalize the literal when the predicate is built and the input before
it is evaluated, with the helpers DataFusion uses for the same purpose.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* test(sqlness): cover flushed float rows holding -0.0

Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`,
`>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of
them, as they do while the rows are still in the memtable.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

---------

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-11-16 00:43:04 +00:00
dennis zhuang 6e025eef61 fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): validate MySQL handshake scope and SQL USE

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): authorize PostgreSQL schema selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): defer PostgreSQL database lookup until authorized

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(servers): propagate catalog lookup errors in protocol tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(postgres): skip unusable search path candidates

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): log database authorization and selection outcomes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): preserve search path responses and schema boundaries

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): clarify database selection diagnostics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): format handshake username in structured logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): secure database-less sessions and failed startup

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): retain authorized default database selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(auth): separate database selection from request dispatch

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(mysql): align schema validation with database-less login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(auth): explain database-less session restrictions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 08:07:57 +00:00
discord9 f88af153ec feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership

Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): send lookup membership as a key list

Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level.

Signed-off-by: discord9 <discord9@163.com>

* docs: tighten narrow membership contract

Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): retain safe bounds on membership fallback

When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): skip unrepresentable dynamic filter snapshots

Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable.

Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): clarify 642-key payload budget comment

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): tighten dynamic membership encoding

Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): reset stale dynamic filters on unsupported updates

Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): scope no-true rule to the encoder

Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): correct reset wording

Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-13 07:51:06 +00:00
Weny Xu 7eb766c9e4 feat(common-meta): make RDS metadata backends closable (#10417)
* feat(common-meta): make RDS metadata backends closable

Standalone closes the metadata backend on shutdown, and
`MetadataKvBackendCreator` now returns a `ClosableKvBackendRef`. Only the
raft-engine backend implemented `ClosableKvBackend`, so external metadata
backends could not be wired into the standalone creator.

Implement `ClosableKvBackend` for `RdsStore` and return
`ClosableKvBackendRef` from `with_mysql_pool`/`with_pg_pool` and the
metasrv RDS builders. The stores only hold reference-counted connection
pools that are released with the last backend handle, so `close()` has
nothing to release explicitly.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): close RDS connection pools on shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): preserve closable RDS constructors and test shutdown semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 07:18:56 +00:00
Weny Xu 180dfc68f7 perf(mito): reduce SST metadata memory overhead (#10356)
* perf(mito): reduce SST metadata memory overhead

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(cache): add a sharded weak interner

Share equal live allocations without exposing lock guards or retaining payloads. Sweep candidate buckets incrementally and cover concurrent publication and weak-reference churn.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): share SST metadata through the sharded interner

Require shared metadata wiring for manifests, canonicalize checkpoint entries without cloning their payloads, and centralize runtime registration. Keep live content variants distinct and verify runtime and recovered allocation identity.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: compile SST metadata consumers across feature sets

Wrap fixture manifest metadata in Arc explicitly and gate the test-only HashMap import with its consuming feature. Validate default mito tests and all cmd targets and features with Clippy warnings denied.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): reclaim retired metadata while idle

Run bounded interner maintenance on the shared blocking runtime without retaining its owner between passes. Shrink sparse indexes and release empty backing storage, preserving live readers and reclaiming their metadata after delayed release.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): use intern-arc for shared SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): expose interner counts and decouple manifest setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(partition): support hashing partition expressions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): hash intern SST metadata without ordering values

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 06:57:23 +00:00
Lei, HUANG ed62f8ff54 feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions (#10361)
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions

`iceberg_hash` partition rules could not be evaluated from SQL, so users had no
way to measure a hash distribution or check which bucket a value falls into.
Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)`
which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)`
as text for copy-paste into `PARTITION ON COLUMNS`.

Both functions are immutable and read-only, and no persisted format changes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): handle untyped NULL and small integer counts

`iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray
that the shared array form had no arm for, so it failed with an unsupported-type
error instead of hashing to NULL. Add a DataType::Null arm in the shared
implementation, so both the SQL function and any future rule evaluation keep the
advertised NULL-to-NULL behavior.

`iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the
value was a valid bucket count, because Signature::any leaves the argument width
unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64
arms, and cover them plus the untyped NULL case in unit and SQL tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_breakpoints SQL function

Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution:
a handful of hot keys can still land in one bucket, and a uniform split keeps
hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes
any number of constant values in any order (one shared hashable type, exactly
the values a partition rule compares the hashed column against), hashes them
through the shared implementation, and prints them sorted and deduplicated:

    SELECT iceberg_hash_breakpoints('iceberg', 34);
    -- 1210000089, 2017239379

The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS`
rule, so buckets can be placed around known key values instead of equal widths.
Columns and NULLs are rejected with explicit errors: a column cannot be hashed
at planning time, and a NULL hash never matches a comparison bound, so a NULL
value cannot be a breakpoint (NULL rows always fall in the lowest bucket).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_ranges_sql SQL function

The helpers so far print bare breakpoint numbers, so a user still had to expand
them into comparison rules by hand: n buckets mean n rules with up to
2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders
the whole list from the same equal-width breakpoints:

    SELECT iceberg_hash_ranges_sql('host', 2);
    -- iceberg_hash(host) < 0,
    -- iceberg_hash(host) >= 0

The output drops straight into the expression list of `PARTITION ON COLUMNS`
(rule per line). It only renders text; it does not create or change partitions.
The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the
rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket
count propagates as NULL; columns, non-string names, empty names, out-of-range
counts and wrong arity are rejected with explicit errors.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): include function identity in ScalarUdf equality

The ScalarUdf wrapper compared and hashed only the signature, so any two
functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash
and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared
equal. DataFusion compares expressions structurally, and its SQL planner
rewrites SELECT expressions against GROUP BY keys with base_exprs.contains
(rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x,
iceberg_hash(x) fail planning, because isnull(x) matched the grouped
iceberg_hash(x) and was rewritten into an aggregate column that does not
exist.

Compare and hash the function name alongside the signature, so functions are
distinguishable while two instances of the same function still compare (and
hash) equal.

Add regressions: same-signature functions stay distinct ScalarUDFs and stay
separable in hash-based containers, and grouping by one while selecting the
other plans and executes in both directions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): quote rendered column names and reject mixed hash domains

`iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a
legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses
as a subtraction; names with spaces, dots, quotes or upper case could not be used
either. Quote the name as a SQL identifier and double embedded quotes, so the
rendered rules stay one identifier and survive parsing. Cover the round trip by
feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain,
dashed, upper-case, quoted and spaced names.

`iceberg_hash_breakpoints` accepted values of different types, but strings and
integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so
mixing them yields breakpoints that describe no real column. Require one hash
domain (all strings or all integers, integer widths may mix) and name the
offending argument otherwise. The existing SQL case that mixed
`('iceberg', 34)` now uses two strings.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(function): register the raw hash UDF and tidy the helpers

Registering `iceberg_hash` through a `Function` wrapper built a different
`ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared
the SQL expression and the rule expression as unequal; the wrapper also rebuilt
a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`,
so register `iceberg_hash_udf()` directly and add a test that the registered
function and `IcebergHash::to_logical_expr` produce equal expressions.

Also: collapse the arity check and slice destructuring into one `let ... else`;
make the bucket-count helpers associated functions instead of constructing a
`default()` instance; pass the caller name through so an error raised from
`iceberg_hash_ranges_sql` names that function rather than
`iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY`
instead of rebuilding and wrapping each one.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): accept every string column name and cover the rules end to end

`iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its
column-name argument, so a valid constant string that arrives as
`Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as
non-string. `Signature::any` leaves argument types unchanged, so the function
has to accept all string representations itself; use `ScalarValue::try_as_str`
and keep the NULL and empty-name checks. A unit case and a SQL-level case pin
that `Utf8View` renders the same rules as a plain literal.

Add a sqlness case that renders the four buckets for the quoted column
`"host-name"`, creates the table from that exact text, inserts one value per
bucket plus NULL, and checks the row counts implied by the rendered predicates
against the rows each region stored. This closes the gap between rendering the
rules and routing rows with them.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(partition): hash constants through IcebergHash::evaluate_array

`iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for
line. Call the existing method from `into_hashed_scalar` and drop the extra
public entry point, so the SQL helpers still hash through the code a partition
rule evaluates without a second identical API.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-13 06:10:13 +00:00
dennis zhuang ca1b8c7c98 fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): redact credentials at their owning fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): cover skipped SAS tokens and export logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask SAS tokens before overlapping account names

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): redact SAS tokens before overlapping account fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask containing credentials before shorter values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 05:35:39 +00:00
dennis zhuang 56cb03f4fb fix(function): handle unaligned binary vector literals (#10380)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 04:07:31 +00:00
Weny Xu 6420082685 fix(meta): skip metric engine identity columns when merging partition columns (#10368)
CreateLogicalTables copies every physical partition column into the logical
schema as a nullable String tag. After a physical metric table is repartitioned
by the internal `__table_id` column, that column reached the region create
request and the metric engine rejected it with "Internal column __table_id is
reserved", so no logical table could be created.

Filter engine-owned identity columns while merging, reusing
`is_metric_engine_internal_column`. Filtering at the merge point also protects
procedures that persisted an unfiltered `physical_partition_columns` list.
User partition columns keep the previous behavior.

Write routing for a physical table partitioned by `__table_id` is still
tracked by #10352; this change only unblocks logical table creation.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-12 12:03:33 +00:00
discord9 ee9ad87fa1 Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main (f8d8fe1071) into narthex (115d85b669) on a throwaway
local branch. 31 semantic conflicts resolved; workspace and all test targets
compile; promql 1215/1215 and mito2 series_index 185/185 pass.

Pre-merge pins and follow-ups are marked with TODO(narthex-merge):
- greptime-proto is temporarily pinned to the fork branch carrying
  GreptimeTeam/greptime-proto#355; replace with the upstream rev once it lands.
- native payload byte cache is disabled (main removed PageRangeCache); two
  samples tests are #[ignore]d until it gets a dedicated page-cache budget.
- columnar metadata now has its own CacheManager tier (a quarter of
  sst_meta_cache_size); the unified decoded tier was not kept.
- query.experimental_native_promql is registered as a query option, otherwise
  the native route is silently rejected by the option snapshot.

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 15:45:04 +08:00
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
2026-11-12 01:09:42 +00:00
discord9 97c8136c3c chore(deps): bump datafusion to ae04160b (#10320)
* chore(deps): bump datafusion to ae04160b

Move all 16 `[patch.crates-io]` DataFusion pins from `cc5534492` to
`ae04160b`, the merge commit of GreptimeTeam/datafusion#42.

That fork commit backports six upstream apache/datafusion fixes:

- #26452 timestamp-to-time casts no longer drive statistics pruning
- #24675 memory-limited NLJ fallback refuses left-emission joins with a
  multi-partition probe side instead of emitting wrong unmatched rows
- #24891 RepartitionExec spill-pool deadlock with two open files
- #25076 null-aware anti join no longer emits rows after a sibling probe
  partition saw NULL
- #25339 correlated NOT IN with a non-equality correlation
- #26578 null-aware anti join drops build rows with a dictionary NULL key

The first three are faithful backports. The last three are re-implementations:
upstream restructured `hash_join/stream.rs` before writing them, so the patches
do not apply to this fork's older copy of that file. `#25339` covers the
single-equi-key plus residual-filter shape; multi-key correlated `NOT IN` still
fails at planning as before.

`Cargo.lock` is regenerated (35 `datafusion*` packages).

No runtime acceptance has been run against this pin; see the PR description for
the three checks worth doing before shipping.

Signed-off-by: discord9 <discord9@163.com>

* fix(common-query): pass the pruning bitmap to HashTableLookupExpr::new

#39 added a fifth parameter to `HashTableLookupExpr::new`; the dynamic-filter
payload round-trip test still constructed it with four arguments, which breaks
`--all-targets` builds even though the library itself compiles.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): bump datafusion to 70f84c5b

Move the pins from `ae04160b` to `70f84c5b`, the head of
`greptimedb-55.1-thin-fork` after GreptimeTeam/datafusion#37 was merged on top of
#39/#41/#42.

`ae04160b` did not contain #37, so the reverse-cast constraint propagation
guard was lost and `tests/cases/distributed/optimizer/partition_pruning_cast`
failed: the filter `CAST(k AS BIGINT) = 1` made the planner treat the float
column `k` as a singleton and drop the requested `ORDER BY k DESC`.

The new pin carries both the reverse-cast guard and the six release backports,
so the regression is covered again.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-10 18:17:24 +00:00
discord9 ffa0815285 refactor(series): hold the frame layout marker at one value
The series frame is the native PromQL path's internal transport, and that path lives on `narthex`, an experimental branch: it may break its own formats between revisions, so a frame revision number has no compatibility promise to protect. The marker becomes movable once the path reaches a release branch, where a producer and a consumer can legitimately be different revisions.

The reader still refuses a frame that declares any other value, which turns a mismatched producer into an error instead of values decoded under the wrong columns. The two wire fixtures are renamed to `v1_metadata.json`, and the drift cases now use the value that is not current.
2026-11-10 16:40:03 +08:00
Weny Xu 65dc999180 feat: support cancelling compaction tasks by task ID (#10229)
* feat: add ADMIN compaction task cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject metadata region compaction cancellation before routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged cancellation commit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 07:38:08 +00:00
Arpit JainandClaude 402e4a97fc fix(time): avoid i32 overflow when converting an out-of-range date (#10299)
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-11-09 16:34:48 +00:00
jeremyhi 50a2c4d378 fix(mito2): publish manifest deltas by conditional create under a lease (#10234)
* fix(mito2): publish manifest deltas by conditional create under a lease

A late old leader and the region's new holder can commit the same manifest
version. For a region whose writes require a lease and whose WAL is the object
store WAL, the manifest now behaves as follows. Every other region keeps its
behaviour and issues no extra request.

- Each delta has one fixed name, `{version:020}.json.gz`, written by a
  conditional create. Without `compress_manifest` the delta is written at gzip
  level 0, so that all writers share the name.
- On an existing object, the writer reads it from storage: identical content is
  a retry of a write that landed, anything else is a retryable
  `ManifestVersionConflict` that keeps the output files.
- The version advances only after a successful save.
- Recovery loads the highest checkpoint found by LIST and uses the checkpoint
  pointer only for its checksum, because a stale writer's checkpoint task can
  leave the pointer at an older version.
- Checkpoint cleanup frees names, so a late create can still succeed on a
  deleted name. A commit whose save returns after the region's Leader deadline
  lists the checkpoints once: a checkpoint at or above its version, or a LIST
  error, is a conflict; otherwise it applies but starts no checkpoint. A commit
  within the deadline does one conditional PUT and no LIST.

`RegionManifestManager::update` becomes the one commit entry and takes
`allow_checkpoint` and `must_verify` as required parameters. A backend that does
not declare conditional create is refused at open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): treat every LIST error as a conflict when verifying a commit

The listing helper turned a NotFound from lister creation into an empty result,
so the check after a late commit could read a failed LIST as proof that no
checkpoint covers the version. List the checkpoints directly, so that any LIST
error reaches the caller. Recovery still treats a missing directory as having no
checkpoint.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the LIST failure in manifest conflicts and follow the style guide

A failed LIST during commit verification was reported as "OpenDAL operator
failed", because the conflict reason used the display of the wrapper error.
Report the root cause through `output_msg` and assert it in the test.

Align the change with docs/style-guide.md: name boolean methods `is_*`, name
the listing helper `list_highest_checkpoint` and move it after the public
methods, qualify `encode_gzip_stored` through its module, and name the mode
setter after the field it sets, which is now `pub(crate)`. Move the shared test
helpers to `crate::test_util` and `manifest/tests/utils.rs` so that the manifest
test modules stay private.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): order the manifest test helpers public before private

Move the private `NotFoundReader` and `CheckpointTaskBlocker::block_write`
below the public items next to them, and document
`fail_manifest_delta_reads_layer`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): fail recovery when a listed checkpoint disappears

Recovery treated a checkpoint that LIST found but that cleanup removed before
the read as no checkpoint at all. It then replayed from the first version over
deltas that were already deleted, and could report an empty region or build a
manifest from the surviving tail. Return a retryable version conflict instead.

Record the size of the stored object when a retry finds an identical delta, as
the other writer may have encoded it differently. Reopen the region in the test
of a commit that crosses its deadline, and reduce the AGENTS.md entry to the
invariant.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): verify conditional commits from callers without a deadline

Restore `RegionManifestManager::update(action_list, is_staging)`. Without a
Leader deadline a caller cannot show that its commit returned in time, so a
normal commit under conditional publication is now always verified on this
path and starts no checkpoint.

Only `ManifestContext::update_locked` has a deadline, so the variant that takes
`allow_checkpoint` and `must_verify` becomes `update_with_verification` and is
crate private. `sst-replace` and the existing tests return to the two-argument
call.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): name the verifying manifest update in the hook docs

`ManifestContext::update_locked` now calls `update_with_verification`, and the
public `update` also calls it, so neither doc can call `update_locked` the sole
caller of `RegionManifestManager::update`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the conditional publication rule in region options

Move the rule that decides which regions publish manifests conditionally
to RegionOptions, so the manifest options only hold the result. Report a
checkpoint that vanished after LIST with its own error instead of a
version conflict, and keep the stored gzip encoder with the other
CompressionType helpers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-09 11:59:33 +00:00
Weny Xu 4e5b60b869 refactor: rename flushed_parquet_bytes to ingested_parquet_bytes (#10258)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 04:25:42 +00:00
Weny Xu 8bdb7d5509 feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 03:17:48 +00:00
fys 3051cb41db feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths

Signed-off-by: fys <fengys1996@gmail.com>

* chore: use rebased proto for JSON path fulltext

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: reuse index target memory estimate for fulltext cache

Signed-off-by: fys <fengys1996@gmail.com>

* fix: filter disabled fulltext options for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* docs: clarify fulltext intermediate path uniqueness

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: name index targets consistently

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): apply JSON fulltext indexes per compatible target

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): skip invalid JSON fulltext index targets

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): require enabled fulltext indexes for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: simplify JSON fulltext index construction and pruning

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): unify fulltext target compatibility and creator APIs

Signed-off-by: fys <fengys1996@gmail.com>

* perf(function): preserve string layouts when evaluating matches

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): centralize fulltext predicate extraction errors

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: remove enable flag from JSON fulltext proto options

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: clarify JSON fulltext index helpers and imports

Signed-off-by: fys <fengys1996@gmail.com>

* fix: allow rebuilt fulltext indexes on ordinary columns

Signed-off-by: fys <fengys1996@gmail.com>

* test: spell out false positive rate in fulltext tests

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto to upstream merged revision

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): initialize fulltext option in multi-region tests

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-06 10:33:10 +00:00
discord9 58a67a47c3 feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait

Signed-off-by: discord9 <discord9@163.com>

* feat(query): transport interleaved raw series fragments

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): reconstruct interleaved raw series at repartition

Signed-off-by: discord9 <discord9@163.com>

* feat(query): plan distributed native raw series

Signed-off-by: discord9 <discord9@163.com>

* chore: satisfy clippy for the raw series transport

The workspace clippy run rejects a redundant rebinding, a constant-size
`chunks_exact`, and an implicit `String` clone in the tests these
commits carry. No behavior changes.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): execute native raw promql over real datanodes

Plan a whole native statement as one fused node over raw series
sources when the frontend reaches regions over the wire, including
single-selector statements, while keeping the in-process split and
segment shapes unchanged. Validate the whole tree with the runtime
constructor before committing to that plan, so unsupported shapes
still fall back to the legacy route.

Bound the region reads of that topology with a query-wide permit
budget of target_partitions, held until each region body is drained;
other plan shapes keep unbounded reads rather than risk a body budget
they could deadlock.

Acceptance on a real two-datanode cluster with TCP frontend clients:
time-partitioned series are reconciled before the fused node and match
the unpartitioned control, with the DoGet peak at the target bound;
series that no region shares keep contiguous per-series order and
reach the fused node without a reconciling router.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): pin greptime-proto to the raw series transport PR

The raw series Substrait message lives in greptime-proto PR #349, which
is not merged, so pin its head commit instead of carrying a local path
patch. Move the rev to the merge commit once #349 lands.

    https://github.com/GreptimeTeam/greptime-proto/pull/349

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): reconcile a discovered union of raw series

A nameless selector reads its matches as one `Union`, and `UnionExec`
reports the frame metadata of its last child. A time-partitioned metric
sorted before a host-partitioned one therefore looked contiguous, and
the fused node skipped the reconciliation of the fragmented member's
series, silently folding region fragments as separate series.

Lower the declaration on the fused edge to the weakest ordering any
member declares, before the fused node reads it. The union itself is
unchanged; the fused node's existing interleaved-input path inserts the
router. A projection carries the lowered declaration because the
pinned physical planner ignores a union's own schema.

Cover it with a planner regression for both member orders and an
all-complete control, and with a real two-datanode acceptance case that
mixes a time-partitioned and a host-partitioned metric and checks the
extrapolated increase per series.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover aggregates over discovered metrics

A nameless selector's metric comes from its own frames, so a folding
aggregate is a multi-input tree even with one selector: apply the same
preflight and order-dependent refusals on the remote route. Cover
`sum(increase(...))` and grouping by the metric name over the mixed
discovery fixture.

A grouping that names only `__name__` stays declined, since the output
schema cannot name that group: here the metrics also differ by `site`,
so grouping by the name and `site` exercises the two groups.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep remote multi-source refusals the split made

Skipping the aggregate split and `segment_tree` over the wire must not
drop their admission refusals. Two sources that fold groups by metric
name would report groups that differ only by name under one label set,
and an order-dependent operation over a discovered selector's union
would answer in arrival order.

Scope both refusals to the actual multi-source inputs rather than to
the logical-table-id heuristic: a discovered selector reads several
metrics through one selector, and a Mito table beside a metric one
carries no logical id.

Cover the Mito-plus-metric grouping refusal on a real two-datanode
cluster, and cover a discovered `sum by (__name__)` over a raw selector
(the name survives it) and a folding `sum(increase(...))`.

Signed-off-by: discord9 <discord9@163.com>

* docs: drop the local proto patch prerequisite

The dependency is pinned to greptime-proto PR #349's commit, so the wire
fixture no longer needs a local path patch.

Signed-off-by: discord9 <discord9@163.com>

* chore: finish the narthex rebase resolution

Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and
pass the engine state through the native planner's callers, while the
remote route still declines the split it cannot carry.

Signed-off-by: discord9 <discord9@163.com>

* test: pass the engine state to the native operand test

The native operand planner now takes the engine state, which decides
whether a statement is planned for remote regions. Keep the histogram
operand contract test on that signature.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep the split and segment plans over the wire

Remote planning disabled the aggregate split and segmentation because
the split's producer read a bare scan. The producer reads whatever the
statement prepared, and over the wire that is the raw series source:
the partial half runs per partition on the frontend, the final half
folds the exported states, and no aggregate state crosses the wire.

Keep the split and let segmentation build evaluated segments from the
raw sources instead of bare scans, so per-series window work keeps the
scan width. Admit the resulting fused chain to the query-wide read
budget, since each fused node polls its input while it is drained.

Remove the recovery-projection wrapper: the analyzer no longer inserts
that projection, and the physical planner consumes the frame boundary
as it is. Serialization is now only the node's encoding: the codec no
longer decides placement, and the metric-name binding is checked
against the declared frames rather than the node types below.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): drop the in-process encodability exception

A split aggregate encodes now, so the analyzer no longer needs a
special case that skipped the encoding gate for it. The gate itself
stays: every plan a region receives still has to encode.

Signed-off-by: discord9 <discord9@163.com>

* test: cover a repeated selector over one partitioned table

Two occurrences of one time-partitioned metric keep their own raw
source, time bounds and identity over the wire, so a ratio of two
windows and a binary expression over two identical ones stay native and
reconcile each occurrence's fragments before the fused node reads them.
Compare both against the unpartitioned control.

Signed-off-by: discord9 <discord9@163.com>

* fix(table): refuse a scan that plans interleaved ordering

A scan's planned ordering is a guarantee its own partitions keep.
Interleaved is the transport annotation for fragments several producers
contribute, so a scanner that planned it has no raw-series read whose
frame would honour that declaration. Reject it in the scan conversion,
while the frame codec keeps accepting interleaved raw frames from the
router that reconciles them.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): assert the metric name beside the value

`sum by (__name__)` over a discovered selector was asserted as a bare sorted
list of values, so a plan that grouped by the wrong metric name - or dropped
the name and folded everything into one series - could produce those same
numbers. Read `__name__` and `val` by column name (the result puts `__name__`
last, and a fold that drops the name drops the column rather than emptying it)
and compare the `(name, value)` pairs against the fixture's own metric names.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep the hybrid route for a partitioned pair

A multi-selector tree over a partitioned table was admitted as one fused node
for any pair of tables, not only for scans of one table. That preempted the
frontend hybrid route, which copies the matcher's labels into each operand's
scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08`
depends on exactly that plan: its partitioned RHS holds a duplicated
`job="other"` group, so the whole-native binary reports the duplicate instead
of the value the hybrid route returns. Refuse the whole-fused plan for that
shape in process again; the remote raw-source admission is unchanged, since a
remote multi-selector tree over a partitioned table is the point of this
branch and the distributed acceptance tests read it.

The hybrid route then has to behave the same way it does without native
operands: `matching_filters::propagate` now runs for them too, and a rewritten
operand is re-planned through `try_plan_native_operand` with its original
`selector_matcher` preserved, falling back to the ordinary plan when the
native route declines. The unpartitioned case, whose duplicate group the
whole-native binary is expected to report, is unchanged.

The two new re-plan call sites put the native route's large future inside the
recursive planner frame at every nesting level (~178 KB became ~209 KB), which
overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where
the new branch never ran. Box the future in a non-async helper and move the
re-plan into `replan_rewritten_operand`, which brings the frame back to the
old size without changing the behaviour.

The partitioned-pair unit test asserted the removed shape. It now asks for the
remote engine state, where a cross-table pair really is one fused node over
two raw sources, and asserts that the same pair in process keeps the hybrid
route.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): pin greptime-proto to the merged raw series transport

GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as
`f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit,
in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an
unmerged ref.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-06 10:31:52 +08:00
discord9 3692222c0f feat(promql): query native histograms through the native PromQL route
Admission reads the field's kind, the output column is typed by what the compiled tree answers, the reader follows the field kind, and the response writes the canonical histogram column. PromQL warnings reach the native kernels.

Tracks #8887.
2026-11-05 19:06:01 +08:00
Weny Xu 4a85e6fcf3 feat(meta): expose datanode drain and scheduling HTTP APIs (#10083)
* feat(meta): expose idempotent datanode drain HTTP APIs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify drain admission and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): reject concurrent datanode drains

Align the drain tracker with region migration and return DrainDatanodeRunning for tracked nodes. Restore the original recover hook and rebuild reservations in the loader without procedure IDs.

Retain admission coordination and recovery readiness to protect scheduling restoration. Update concurrent submission and persisted recovery tests for rejection semantics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): allow procedure recovery retry after startup failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): classify active drain conflicts as illegal state

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 08:34:13 +00:00
Weny Xu d750d25965 perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: push down SST manifest scalar aggregates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: remove redundant SST manifest inspection guide

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: generalize SST manifest aggregate pushdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: retain frontend aggregation for unsupported inspection states

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: expose remote SST aggregate plans in EXPLAIN

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: normalize SST analyzer in TQL explain snapshots

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: guard SST predicate types before remote inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:37:00 +00:00
Ning Sun b21b1cdb69 fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-04 18:18:33 +00:00
discord9 65a89945dd feat(flow): support caller-owned fenced incremental deltas (#9996)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
2026-11-04 18:18:23 +00:00
discord9 fe2e364d23 feat(promql): execute split native aggregates as partial state over a state router (#10147)
* feat(datatypes): add the shared aggregate-state representation

A frame that carries partial aggregate state has to say which aggregate it
is state of and which accumulator components it carries, or a consumer can
read a sum state as an average one. These types are that statement: the
kind, the ordered component list, one position's values, and the schema
that validates the two against each other.

Nothing uses them yet; the frame content that carries them follows.

Signed-off-by: discord9 <discord9@163.com>

* feat(datatypes): carry aggregate state as a third frame content

A state frame keeps the series-major columns a frame already has — the
key, finality, grid spans and labels — and puts each aggregate field's
accumulators in one list of component structs under `fields`, the shape a
classic histogram payload already uses. An absent list item is that field
having no contribution at that position, so a hole stays a hole instead of
becoming a zero, and a group state — which carries no component at all —
still says per field where it was folded.

The reader keeps typed handles into the batch rather than expanding the
state into rows and rebuilding it, so the merge above it reads the arrays
where they lie.

Signed-off-by: discord9 <discord9@163.com>

* feat(common-query): pack and unpack aggregate-state series frames

The packer takes what a partial aggregate produces — one group, its carried
positions and its per-field state — and the unpacker hands back the row with
typed handles into the batch. That asymmetry is deliberate: export
columnarises once, import must not rebuild the state to read it.

Validation mirrors the evaluated pair: the declaration is parsed, contiguity
required, the empty batch decoded before the stream is returned, and kind,
component order, grid and field count are each checked with the part that
disagreed named.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): run native partial and final aggregates

An aggregate node used to compute only `Complete`, refusing the stages a
split aggregate needs. It now folds evaluated chunks into kernel state for
`Partial` and merges that state into values for `Final`, so the compiler's
`Partial -> Final` plan has a runtime behind it.

The executor gains a third chunk kind: an input leaf accepts raw, evaluated
or state chunks according to what its binding produces, never guessing from
a schema, and a state chunk cannot be handed to a node that reads values.
The state binding now carries the aggregate kind and its ordered components,
derived from the kernel's own state fields, so a sum state cannot be merged
as an average one.

The sibling router refuses aggregate state explicitly for now; the node that
routes it follows in its own change.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): route aggregate state by its group labels

A state row is one group, complete as one producer wrote it: there is
nothing to chain and nothing to merge, so the existing repartition node —
whose invariants are all about assembling a key's chain and waiting for
every input to close — is the wrong place for it. This node does one thing:
send each row to the output its canonical labels name, so every
contribution of a group reaches one owner and the merge above sees all of
them.

The read belongs to the outputs that were claimed, not to the plan, so
dropping every output stops it and returns what it reserved.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): execute partial aggregates as state frames

A fused node now reads its inputs by what they produce — raw samples,
evaluated values, or aggregate state — instead of inferring the kind from a
schema, and a partial root answers state rather than rows: the executor's
state chunks are packed into the aggregate-state frame the merge above
consumes.

The temporary partial-fold bridge goes with it: no marker field, no side
channel fold, no compatibility accessor. A tree that folds on the side is
exactly what this change exists to remove, and the aggregate merge that
replaces it is a separate node.

The state path derives its mode and its binding from one reading of the
root, so the two cannot disagree, and no production path panics on a
malformed tree.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): split native aggregates into partial and final trees

The previous commit made a `Partial` aggregate export its state, but left
`AggregateFinalize` in place: it still called `FusedSeries::try_new_partial`
and `partial_fold`, which that commit deleted, so the crate did not compile.

Replace the bridge with the two trees that run a split aggregate:

- `native/split.rs` owns the cut. It moves out of `aggregate_finalize.rs`
  unchanged in structure, and `Split::into_trees` binds the two halves: the
  prefix under a `Partial` aggregate answering aggregate state, and a `Final`
  aggregate consuming that state with the suffix above it.
- The suffix is narrower than the old merge admitted. Only point-wise maps and
  a binary operation with a constant side may sit above the cut; `topk`, the
  classic-histogram functions, a second aggregate and a vector matching are
  declined, because the groups a `Final` aggregate owns are split across
  partitions and each of those reads a whole vector.
- `FusedSeries` gains `try_new_state_export` (a `Partial` node declares its
  own state frame schema rather than taking a caller's), `split_aggregate`
  (declining shapes become `None`, not an error), `reads_aggregate_state` and
  the state-input path of `to_execution_plan_with_parallelism_option`, which
  routes state through `StateRepartitionExec` and lets a cross-series tree read
  state from more than one partition.
- The runtime executes the `Final` stage: the guard that refused it is gone.

`AggregateFinalize`, `aggregate_exchange` and their tests are deleted; nothing
in the crate referenced the exchange module.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): route aggregate state through the series repartitioner

Replace the dedicated state-repartition node with the existing series
repartition execution: state batches are routed by their canonical labels
without being rebuilt as StateSeries, and each row slice is charged for the
Arrow wrappers it newly allocates while the parent buffers stay charged once.

Carry a producer's state charge with the chunk it hands over, so the exporter
that packs rows across polls remains accountable for the batch it still holds
instead of releasing it at handoff.

Read a Final aggregate's value and tag columns positionally, which is the
order the split's output schema is built in, and name them through the
legacy expression only at the outer boundary; a bare sum over a metric whose
value column is named after the whole expression no longer fails planning.

Pack aggregate-state frames in one pass over positions and validity instead
of scanning the contributing rows for every position.

Cover the changed behaviour with execution tests for bare global and grouped
sums over one- and two-value-field tables, for the handed-over state charge,
and for the export lifecycle on drain and early drop.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): bound tree depth on both sides and validate merge inputs

The depth limit was decode-only and set to a number nobody had measured. A
129-deep tree could therefore be built, serialized and shipped, while the
receiver refused it; and in CI the boundary test itself aborted with a stack
overflow, because a 2 MiB thread was asked to build, serialize, decode and
drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is
not covered by a guard that only wraps `Deserialize`.

Measured instead of guessed. A probe walked build, serialize, decode,
validate and drop at increasing depth inside a named 2 MiB thread: 32 through
128 completed, 160 aborted. The cap is now 64, half the highest passing depth,
and the same number is enforced in `NativeTree::try_new` and in the
deserializer. Verified through the real constructor: a 64-node tree
round-trips, and a 65-node tree is refused when it is built as well as when it
is decoded, so sender and receiver accept the same set. The boundary test runs
on a named thread, which is how a future failure will identify itself.

The merge node accepted any single input. `with_exprs_and_inputs` rebuilt
`AggregateFinalize` over whatever it was handed, on the reasoning that a
rewritten input only changes where the state comes from. Nothing checked that
it was state at all -- and until now nothing could produce a counterexample,
because a values-producing producer could not be serialized. The serializer
was standing in for a validation that belonged to the node. `with_exprs_and_inputs`
now rejects an input whose schema is not `state_schema(binding.tags)`, so
substituting a computed-values producer fails at the logical node with a
message naming the expected schema, rather than depending on an encoding
limitation. `AggregateFinalizeExec` already had the same check at physical
planning; the serializer stays permissive.

The three placement tests in `commutativity.rs` asserted the old refusal as
their mechanism -- an ordinary fused node "fails to encode", a partial
producer "cannot be pushed down". Those expectations were the encoding
limitation wearing a test's clothes. They now assert the responsibilities that
remain: an ordinary fused node round-trips through the encoder, a partial
producer rides to the region while its finalize stays on the frontend, and a
values producer handed to a merge is refused by the schema contract.

The test-only `NPRQ` envelope is gone. It framed trees with its own version, a
1 MiB cap and its own rejections, none of which production had, so
`native_tree_envelope_rejects_version_size_trailing_and_truncation` was
proving a helper's protocol while the PR description claimed a production
guarantee. Trees are tested through plain postcard, transport through the real
`FusedSeries` interface. The redundant clone-and-revalidate in
`FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already
routes through `try_new`.

Signed-off-by: discord9 <discord9@163.com>

* style: let taplo order the workspace dependency entry

`taplo format --check` runs over the whole workspace in CI, not over the two
manifests I had checked. It sorts the `[workspace.dependencies]` table, so
`postcard` belongs in alphabetical position inside the
external-dependencies block rather than immediately after its comment. Local
checking was narrower than the CI command, which is why this only surfaced
there.

Signed-off-by: discord9 <discord9@163.com>

* style: place postcard in the sorted position the review asked for

Inside the external-dependencies block, `taplo` was willing to leave the entry
before the `axum` comment, because a comment line ends a reorder group. The
review asked for the alphabetical position, which `taplo format --check` also
accepts, so use it.

Signed-off-by: discord9 <discord9@163.com>

* docs(promql): name the real reason a multi-source name grouping declines

The comments still said the fused node's declared columns cannot name
`__name__`. Since #10106 a discovered source declares it, so a nameless
selector's `sum by (__name__)` survives; what actually declines is a tree
reading more than one *named* source, where the table name is the metric
name and no source declares the tag. Same conditions, accurate reason.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): find the topk producer by behaviour, not by node count

`native_aggregate_topk_over_global_partitions` asserted that the whole plan
holds exactly one fused node. Once native frames cross the region boundary a
plan legitimately holds the frontend root and the region producers at the
same time, so that count is no longer an invariant.

The test now selects the segmented producer and the ranking node by what
their plans do: at least one producer contains `segment` and none of them
runs `topk`, and exactly one node across the fused and final captures ranks.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-04 21:25:41 +08:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
Minh Vu 25003cea16 fix(time): parse zero-hour timezone offsets correctly (#10091)
fix(time): preserve zero-hour timezone offset signs

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
2026-11-04 06:49:45 +00:00
discord9 b30557960b feat(promql): preserve scan parallelism through native frame boundaries (#10055)
* fix(promql): let a closed chain keep its charge until the packer is done

Four places released the frame reservation: the early-publish path handed it to
the packer, but the queue drain and the end-of-input path freed it in place. Those
two run while the packer still holds rows it has not turned into batches, so a
series spanning several batches went unbilled once its last piece was handed over.

Every emit path now hands the charge over, and the packed stream frees what it
holds: what is left when it is polled and pending is the one identity it is
turning into batches, and nothing survives its end or its drop.

The identity guard is also widened: within one source every series shares its
metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is
what a range function, a math map and the timestamp shape do -- still maps
distinct series to distinct labels. Only rewriting an ordinary label can make two
equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame
output therefore publish as their chains close instead of buffering a partition.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must

Three defects in how this node decided a tree's execution layout.

A match whose two operands derive from the same source still needs its match keys
co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)`
therefore fell through to single-partition execution and lost the parallelism the
scan gave it.

A raw input never received a matching router: the normalization branch returned it
unchanged unless the whole tree was global, while the builder requires a direct
matching router for any multi-partition matching-local input. A two-source raw
query was rejected at planning.

Nothing reconciled the partition counts of different sources either, though the
builder requires one common positive count. A single count is now computed for the
matching-local case and every router under it uses it, gathering when that count is
one.

The window restriction also ran before normalization decided the layout, so a tree
about to be gathered could be refused for a limit that only applies to partitioned
windows.

Two tests changed with the behaviour rather than around it: hash routing does not
promise which partition holds a given series, so that test compares results
instead of placement while keeping its partition count, per-partition execution
and operand-order assertions; and a test that pinned the old refusal of a
multi-source frame now pins its acceptance.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): keep a raw source's scan partitions instead of gathering them

`RawSeriesSourceExec` declared one partition and read every child partition
inside `execute(0)`, so a cross-table node lost the scan's parallelism before
its inputs reached it. It now declares the child's own count, executes exactly
the partition it is asked for, and re-labels that partition's frames.

That leaves `SeriesGather` with no consumer: it existed to interleave several
partitions into one stream. The module is gone rather than kept as a hidden
N-to-one path a later plan could quietly fall back on.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): publish a raw source's frames under the ordering it declared

`pack_raw_series` derives the frame schema from the stream's own ordering, and
`RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing
the scan's stream through unchanged broke that agreement: a columnar scan
promises key-then-time, so the region published key-then-time frames while the
frontend still expected the schema it had built, and `MergeScan` refused them.

The forwarding adapter restates the ordering this node declares. It is the
weaker promise, so a scan that keeps a finer one still satisfies it, and every
other stream property is forwarded unchanged.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw frame that names its labels by its key

A region cannot hand out complete labels: it has no metric-name binding, so
`prepare_raw_scan` refuses an explicit binding outright. What it hands out is
the stored key plus the description a decoder needs, which is the normal input
to this node rather than a case to refuse — every single-table selector is
planned that way. The node's constructor refused exactly those frames, so any
plan that inserted a router under a single-table selector failed before it ran:
the matching policies this branch adds, and the one-partition normalization of
an evaluated tree.

The node publishes labels, because that is what it groups and closes identities
by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the
input is decoded once, at the unpack boundary this node already has, and an
already-labelled stream passes through it untouched. The frame it declares is
rebuilt from the input's own declaration instead of being compared against it,
which keeps the check that a frame this node cannot re-declare is refused while
letting input and output differ in how they name the same series. `frame_schema_with`
becomes public for that rebuild, since it is the one place that decides what a
declaration is.

`FusedSeries` now describes the router's output rather than the scan's when it
wraps a raw input, because the binding it hands `build` has to match the frame
the router actually publishes.

Tested through the node directly: a key-labelled input routes at all; matching
puts both pieces of one label set in one owner and closes it once; and
`RawPrimaryKey` still hashes the key rather than the decoded labels.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): measure a routed pair by the width it routes to

Four assertions still described the layout this branch replaced: a node with two
inputs used to read every partition of both into one and answer one partition,
and that gather is gone. The pair is now matched in the partitions it was
scanned in, so the fused node answers as many partitions as its inputs route to.

Each of these fixtures scans with four partitions, and on the base of this branch
all five tests passed, so this is a behaviour change this branch made and did not
re-verify rather than a defect: the value assertions behind these shape
assertions pass unchanged, in both the native and the legacy comparison.

The width is read from the routed children instead of being written as four, and
the scan is found by walking down through the nodes that only pass its frames
along, since a routed input is no longer a bare `SeriesScanExec`.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): normalize a multi-partition series-frame root

A query engine hands its plan to one stream, and for a plan with several
partitions that means `CoalescePartitionsExec` - ordinary batch coalescing.
A series frame cannot survive it: one stored series may be split across
partitions, and a chunk's `is_last` closes that producer rather than the
series, so interleaving two partitions' batches produces a stream whose
decoder refuses it (`raw series key changed before final chunk`). Evaluated
frames have the same shape of problem: their validator keeps one open chain,
and a batch from another partition interrupts it.

The engine's root is also a region's root, which is how a cross-table native
statement reaches this: its per-region sub-plan is rooted at a partitioned
`RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end,
before the final sanity check, and it normalizes the root and nothing else -
`FusedSeries` already inserts its own owner wherever a consumer needs one.

A single-partition root is left alone: one producer is the whole answer. A
key-labelled raw root is refused rather than coalesced, because the router
checks that one stored key decodes to one label set and it needs the labels to
do that. No production plan reaches that rejection today - the only producer of
a key-labelled frame is the region's series scan, and every plan that executes
it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish
explicit labels - and the rejection is what keeps it that way.

`schema_check` stays enabled: for the roots this rule accepts, the router
re-declares its input's frame with the same content, label mode, decode
description and ordering, so the schema it publishes is the one it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): admit an evaluated frame as a fused input

The extension planner hands a child straight to the fused executor when that
child already publishes a series frame, and until now it only recognised a raw
one. A segment's output is an evaluated frame, so a statement planned as two
segments could not be bound above the first one.

The check it replaces asked for a raw frame; the property that matters is that
the frame names its own series, which is what `labels() == Explicit` says -
evaluated frames always declare it and carry no decode description. Everything
else stays with `FusedSeriesExec::build`, which already refuses a mixed-content
node, an evaluated input that is not contiguous, and a frame whose declaration
does not match the binding it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): plan a global topk under an aggregate as two segments

`sum(topk(1, m))` needs the whole vector for its ranking, which is a property of
the tree rather than of the statement, and the routes that existed could only
answer a whole-vector tree in one partition: the statement fell back to the
existing planner. Splitting it lets the per-series prefix keep the scan's
partitions while the ranking sees every candidate.

The cut goes below the `topk`, not below the `sum`. A segment holding a `topk`
reads across series, so it would gather its own input before evaluating the
prefix - correct numbers, no parallelism, which is the thing the split is for.
Below the `topk`, the prefix runs on the scan's partitions and the ranking and
the fold above it run once, on the child's evaluated frame.

Both segments are built from one compiled tree: the child is the prefix subtree,
the parent is the original tree with that subtree replaced by an input bound to
the child's frame. Nothing is re-parsed, so the grid, the matching rules and the
bound scans are the ones the compiler produced.

The parent needs its output columns without asking the existing planner to name
a computed column after its expression, so `fused_output_tags` now derives the
grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels
through. It derives output *columns*, not a complete runtime label set: the
metric name is bound separately from the tag columns, and a grouping by
`__name__` keeps that name in the labels while contributing no column.
`fused_root_schema` is that derivation, shared with the ordinary route.

The segmenter is its own method, boxed at the call site. Inlined, its future
made every frame of the planner's recursion large enough to overflow the stack
on a deeply nested statement - `test_parse_or_operator` aborted before this.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the two-segment topk plan and what it keeps

The statement's two segments do not land in one physical tree: the prefix has
one scan for a child, so the categorizer classifies it commutative and pushes it
into the region, while the aggregate above it stays at the frontend. The
observer therefore records every plan a query engine wraps, not only the first,
so the test can look at the region's own root.

What the assertions pin, and why each one is worth its brittleness:

- the answer is the global maximum, not a sum of per-partition winners;
- the frontend segment holds the complete sum and the global topk over an
  evaluated input, with no instant evaluation left in it;
- the region's segment holds the prefix and its output is an evaluated,
  explicitly labelled, contiguous frame;
- that segment keeps more than one output partition, which is the whole point of
  cutting below the topk;
- the region's root is the series-aware router, one output, and it reads that
  segment.

Column names are not compared: this route names a computed column after its
compiled root while the existing planner names it after the expression, which is
older than this split and not what this test is about.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): share the input-locality classification

A tree's distribution requirement is one property, and two layers now ask it.
The physical planner already decided with `input_locality` what a tree needs
from its inputs; the logical planner has to ask the same question when it splits
a statement, or the two can disagree about what a subtree requires.

`InputLocality` and `input_locality` are therefore public within the workspace.
The classification describes the external inputs a tree needs - not its output
distribution, and not which neighbouring operators may share a node - so a
caller that reads it as either of those will cut where it should not.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement where a gather would swallow its prefixes

The previous commit planned `sum(topk(k, prefix))` by matching that shape. A
planner that grows one matcher per query shape grows one per benchmark entry,
and the shape is not what decides where a statement divides: the tree already
says what each subtree needs from its inputs, and `input_locality` answers it.

So a segment is now a maximal subtree with one distribution requirement. Inside
a tree that must gather, any child that (a) can cross a boundary - it produces
evaluated samples rather than raw samples, aggregate state or a scalar - and
(b) does not itself need a gather, becomes a segment of its own and keeps the
partitions it was scanned with. Nothing else is cut, and a tree with nothing to
cut keeps the plan it had.

Every segment rebinds its own inputs: the scans it reads directly first, in the
order it first names them, then the segments cut out of it, because
`FusedSeries` requires dense source indices over its children. `fused_output_tags`
derives the parent's columns from those inputs rather than from the statement's
first selector.

Three gates keep this from answering what the routes before it would not. Each
one is a failure this change produced before it was added:

- the existing planner must still be able to plan the statement. Deriving tags
  locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` -
  an empty range, which the existing planner rejects - was answered instead of
  failing;
- every segment's tree must construct a `NativeExecutionState`, because nothing
  below can fall back once physical planning starts, and a grouping by a value
  field is a shape the runtime refuses;
- the tree must contain a prefix that a boundary could carry, so a whole-vector
  tree with no per-series work inside it is not split for nothing.

The scope is one selector. A statement over two tables binds each scan as a
`RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in
a region, so its prefix would not gain the partitions the cut is for; those
statements keep the plan they had, which the acceptance test pins.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the segment boundaries and their scope

One statement per shape the cut can meet, asserting what the rule says rather
than what one query happens to produce: a per-series tree stays one node, a
matching tree routes instead of gathering, a global fold over per-series work
cuts that work out, a constant stays embedded because a boundary cannot carry
one, and a statement over two tables keeps the plan it had because a segment
over `RawSeriesSource` cannot be placed in a region.

Answers are compared as sets of rows with the value column renamed. Three
things differ between the routes and none of them is the answer: this one names
a value column after its compiled root while the existing planner names it after
the expression, the two emit columns in a different order, and this one is
partitioned so its rows arrive by partition. Comparing them as what they are
leaves the values as the only thing an assertion can fail on.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep a selected segment whole

A segment is chosen because its subtree needs one distribution; cutting inside
it again contradicts the choice. The inner builder was still applying the cut
predicate, and that predicate is "exportable and not `SinglePartition`" - which
every node inside a per-series subtree satisfies, since a per-series tree has no
`SinglePartition` node in it to begin with.

`sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one
segment, then `abs` was cut out of that segment, leaving a middle segment whose
child is a node rather than a scan. That middle segment cannot be placed in a
region, so it answered one partition while the scan had several - the prefix
lost exactly the parallelism the cut is for.

The inner builder now collects the segment's `Input` leaves and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): bind a segment to its own scan

A cut segment carries one source's identity: its scan, its metric name, its
time index, and its labels. Building it from the statement's first selector
only happens to be the same thing while a statement has one selector.

`build_inner_segment` takes the two lists the planner already has - each
occurrence's filtered scan, and each occurrence's metric, time index and tags -
and binds the segment's single source from them. The parent keeps the wrapped
children and the statement-level binding it has now, because a source that
reaches the parent directly still has to cross a region as a raw frame.

A segment that collects more than one source is declined. `MatchingLocal`
satisfies the cut predicate while reading two sources, and one segment cannot
carry both sources' identity.

No behaviour changes: the entry is still one selector, so no scan is wrapped
and the statement-level values are already that selector's.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement that reads more than one table

A whole-vector statement over two sources is now cut into one per-series
prefix segment per source, with the fold above them in the frontend. Each
segment binds its own source's scan, metric, time index and labels, so it can
be placed in the region that holds that source, and each prefix keeps the scan
partitions it was read with.

Three things had to change for the existing segmenter to reach that shape:

The cut sat inside `!per_partition && !multi_input`, so a statement over two
metric tables - which is `multi_input` - never reached it. The cut now sits
under `!per_partition` alone. The aggregate split keeps its own conditions,
including `!multi_input`, so a multi-input tree is still not split that way.

The single-selector gate was doing two jobs. A statement naming several
selectors declines unless those selectors read more than one table: the count
alone says nothing about whether the tree has more than one input to place, and
the shapes it would otherwise admit - one metric named twice under a fold, such
as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers.
`cross_table` rather than `multi_input` decides this, because two Mito tables
carry no metric logical id and are still two sources.

A segment that collects more than one source is declined. A matching subtree
over two sources is a cut candidate, and one segment cannot carry both sources'
identity, so `sum(topk(1, m / n))` keeps the plan it had.

The segmented parent reports the value column the compiled tree names, not the
one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where
the existing plan reports an expression-shaped name. The values are identical
and both names are pinned in the test, so a change to either rule shows up
there. Restoring the existing name needs a rename above the parent - the node's
own schema must keep the compiled names its output columns are classified by -
and is left to a follow-up.

Signed-off-by: discord9 <discord9@163.com>

* test(query): pin that two sources keep their own scan bounds

A segmented statement binds each prefix to its own source's scan, and A1 made
that binding: `bare_scans` is derived from the prepared scans after each
selector's time filter has been pushed into its own `TableScan`. Nothing tested
that the two segments actually read `their` bounds rather than one window applied
twice - the existing two-source case has no offset, so both sides read the same
range and a crossed binding would answer the same numbers.

`sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only
in the right side's window, so the two queries must disagree: 20/1 and 30/2
against 20/2 and 30/3. The test asserts they do, which is what fails if both
segments were given the left scan's bounds.

The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints
only the peers: the debug form carries the table each segment reads and the
embedded `FusedSeries` segment. An in-process region hands the plan observer no
separately wrapped region tree, so the per-partition evidence for segments is
measured in `native_aggregate_test.rs::check_two_source_segregation` instead.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): decline a multi-source tree that groups by the metric name

The gate that lets a multi-source statement reach the segmenter is
`selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above
it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`.
Two ordinary Mito tables differ in table name and carry no metric logical id, so
they are `cross_table && !multi_input`: the guard did not apply and the statement
reached the segmenter.

`fused_output_tags` keeps a grouping by `__name__` out of the declared columns on
purpose - the name travels in the chunk's labels, not as a tag column - so
`sum by (__name__)` over two sources derives no tag column while the runtime
holds two groups that differ only by metric name. Both are then reported under
one label set, as two rows a reader cannot tell apart. That is the failure the
metric-logical-table test already documents; it could not catch this path because
its tables make `multi_input` true.

The check is scoped to a tree that reads more than one source. One source folds
one metric, so its groups carry the same name and naming that name or dropping it
cannot merge two of them.

Signed-off-by: discord9 <discord9@163.com>

* test(query): cut a range function on each side of a division

The two-source cases so far cut an `Instant` prefix: one sample per grid point.
A window function's prefix computes its window inside the region over that
source's own samples instead, so the segment carries a `Window` rather than an
`Instant` and nothing tested that such a prefix is still a cut candidate.

`sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step
the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then
4/3. Two different step values are what let a shared window bound change an
answer, which is asserted before the result is compared.

The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset
case: an in-process region hands the plan observer no separately wrapped region
tree, so the per-partition evidence lives in `native_aggregate_test.rs`.

This is the shape of the throttling query in the benchmark corpus: a window under
a fold, on each side of a division.

Signed-off-by: discord9 <discord9@163.com>

* test(query): plan the benchmark's throttling and namespace-ratio shapes

Every two-source case so far divides two ungrouped scalars. The corpus queries
that motivate this work group by a label on each side, so each segment reports
several groups and the parent matches the two sides by that label. Nothing
tested that path.

`container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total`
is the corpus query verbatim: two metric logical tables on one physical table,
`sum(increase(...)) by (container)` on each side. It plans as one frontend
`FusedSeries` over two `MergeScan` children, each holding a segment bound to its
own metric, and reports the same labelled rows as the existing plan.

The namespace-CPU-usage-over-requests shape differs from the corpus text only in
the metric names, because the corpus names carry recording-rule colons that a
table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form
and is asserted as that shape rather than as the query verbatim.

Both tests assert the native route answered, so a decline fails here instead of
passing on the legacy rows. The value column still differs from the existing
plan's - the same known gap recorded for the ungrouped case - and both names are
pinned.

The fixture helper records the three setup traps the hard way: `INSERT` needs
explicit column lists because the physical table's column order differs from the
logical table's, SQL keywords used as tags need quoting, and no sample may sit at
timestamp zero, which both the scan's lower bound and PromQL staleness drop.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record which native route every corpus query takes

The corpus is 29 queries and nothing said which of them the native route
answers, or by which of its plans. One parameterised test now runs all of them
over generated fixtures and records, per query, the route and the numbers.

Four routes, and the frontend tree alone cannot tell them apart:

- `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one
  per source; the prefix work happens in the regions.
- `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region
  `FusedSeries` that reports `partial state:`, so what crosses the boundary is
  aggregate state.
- `region-local` - the frontend holds no native node at all: a per-series tree is
  commutative, so the whole `FusedSeries` is pushed into the region. A classifier
  that only looks at the frontend calls these legacy, which is what happened
  before this test existed.
- `legacy` - no native node on either side.

All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two
`region-local`, none legacy. The two `region-local` ones are the corpus queries
that fold nothing - a bare selector and a `rate` - which is also why pushing the
whole tree into the region costs nothing.

The test asserts only that the existing plan answers each query with rows, and
that a native route's labelled rows agree to within `1e-10` relative. It records
the route rather than asserting it, because which route a shape takes is a
property of the planner that this test exists to report, not to legislate.

Ten queries differ from the existing plan in the last bits; the largest relative
difference is 2.9e-16, which is summation order, not arithmetic. The table prints
that figure per query so a future equality failure can be told apart from
rounding without re-measuring.

Every fixture here is generated: explicit `INSERT` column lists, quoted SQL
keywords, and no sample at timestamp zero. They establish the plans and the
values; they say nothing about speed, memory, or how many partitions actually
produced rows.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record how many partitions each corpus query's nodes used

The route matrix said which plan answers each query. It did not say whether that
plan produced anything, so a node that ran and emitted nothing looked the same as
one that worked.

`output_values_per_partition` reads `native.root.output_values` off a
`FusedSeriesExec` and returns the counts per partition, and the matrix prints them
as `partitions used`. `assert_four_native_partitions` keeps its own assertions and
now builds on the same helper.

What the column does not establish, and says so where it is printed: these
fixtures hold a handful of series each, so a scan puts them all in one partition -
every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the
plan. The one place a fixture is built to fill the scan's partitions is
`native_aggregate_test.rs::check_two_source_segregation`, which asserts all four
produce values for both segments of a two-source statement. This matrix records a
single assertion instead: at least one partition produced values, so a node that
ran and emitted nothing fails.

Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise
divisions. Their region side has no `FusedSeriesExec` at all, because both scans
are matched inside one region: that is the route, not a missing measurement, and
the table says `no metric` rather than implying zero.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record the new rule in the explain stages

`TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering
`SeriesFrameOutput` adds one line to that listing whether or not the rule changes
the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary
tables, whose schemas carry no series frame declaration, so the rule returns the
plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the
existing lines say for the same reason.

Regenerated by the sqlness runner; the diff is those three lines and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* test(query): make the corpus matrix prove what it claims

Three gaps, all in the test the "every corpus query answers natively" claim rests
on.

The partition helper returned early on a plan that reported no
`native.root.output_values`, so a producer that registered nothing passed on the
declared partition count alone. It now expects the metric: the point of the
assertion is that every partition executed, and an absent counter says nothing
about that.

`segmented` was decided by the frontend tree containing a `MergeScan`, which also
matches a frontend native node consuming raw frames from its regions - a
different placement that says nothing about a prefix running in a region. The
merge scan's `Debug` carries the plan its region runs, so the check now reads
that for a segment; the raw case gets its own `frontend-raw` route. Four queries
move there: L05, L06, L08 and R03.

The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been
probed in a throwaway run. They are entries now, with the join labels Q08 and Q09
match on and real `le` buckets for Q10's histogram. A fallback is no longer a
route to record: a case that reaches the existing plan fails, because that is the
regression this table exists to catch.

Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): let the compiler decide which calls this route answers

A second list of function names stood beside the compiler's own capability
check, and the two disagreed. `admitted_selector` named five calls; the compiler
folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and
more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same
call meant two different things depending on what wrapped it.

The compiler already answers this question: `compile_call` refuses a name it has
no kernel for with `NotImplemented`, and the planner turns that into a decline.
With the list gone there is one place that decides, and a call the compiler folds
reaches the route whether or not it is wrapped in an aggregate.

`sort` and `absent` read across a vector rather than one series at a time, which
is why the old list named them out. They compile, so they are planned; the tests
that asserted the refusal now assert the admission. Only the shapes the compiler
genuinely cannot express keep the existing plan - a subquery evaluates its body on
a grid of its own, which this route does not offer.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): admit an operand by its shape, not by call name

An operand of a hybrid binary was admitted only if it was a selector under one of
five named calls. The compiler folds many more than those five, so `ceil(m)`,
`irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under
an aggregate was admitted - the name list disagreed with the compiler about what
this route can answer.

What the list was really protecting is a layout assumption, and the shape states
it directly: `try_plan_native_operand` hands back a selector's planner context,
and the hybrid caller reads field columns and tags off that context, so an
operand whose output layout is not its selector's cannot be one. `sum(A) / B` is
refused for that reason, not because `sum` is a name.

`single_selector_spine` states it on the expression: a selector, parentheses,
unary, a call carrying exactly one vector-or-matrix argument, or a binary whose
other side is a compile-time constant. Calls are followed by the type of their
arguments rather than by `constant_value`, because a call's string parameters are
constants that `constant_value` does not fold and would otherwise look like data.

Which calls this route answers is now the compiler's answer alone: a name it has
no kernel for comes back as `NotImplemented`, which the planner turns into a
decline.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): mix raw and evaluated inputs in one fused tree

`build` refused a tree whose children were not all raw or all evaluated,
but nothing below that point assumed uniformity: `execute` picks the
decoder from each source's own schema, `FusedFeed` carries the source
index, and `collect_source_schemas` already validates a repeated source
against its own grid. A frame stream has one format per stream, not one
per consumer, so one segment can decode the raw samples of a rate and
compare the result against a vector a previous stage already evaluated.

Remove the refusal and make the remaining per-input decisions per source:
`evaluated_global` now asks about the source it orders, and
`window_sources` keeps only the evaluated sources, since a raw source is
answered by the raw path it always used.

Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y`
arrives evaluated on the statement grid. The test asserts all ten
(host, timestamp) rows and fails on the removed refusal if the refusal is
restored.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): bind a segment to every source its subtree reads

`cut_as_segment` accepts any exportable subtree that is not
`SinglePartition`, so a matching subtree over two sources - `m / n` - is
a legal cut point. `build_inner_segment` then required exactly one source
and returned `None` for it, and `segment_tree` propagates that `None` as
"the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back
to the legacy path even though its cut was legal.

Build the segment from every source the subtree reads. Each child gets the
`RawSeriesSource` boundary already used for separate metric tables, which
carries that source's own metric, tags and fields across the boundary and
makes the child placeable; a segment over two tables cannot sit inside
either region, so it runs above them. The single-source case keeps its
existing direct-scan path unchanged, which is what lets a one-table
segment still be pushed into its region.

Field names come from `tree_source_fields` on the rewritten tree, whose
source indexes are already dense in the order the children are built. That
helper also validates a repeated source against its own schema and refuses
a non-dense index set, which a second traversal would have to re-derive.
It becomes `pub` and is re-exported for the query crate to call.

Take the segment's time column from its own first source rather than from
the statement's first: a cut whose sources are `[1, 2]` would otherwise
name the time column of a table it does not read.

Cover the construction with a planner test and with a real-Mito
integration test that asserts both the exact values and the route: one
segment over the two tables, each child reading its own table through its
own raw series source. Without the route assertion a silent fallback would
pass on equal values.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): run a matching tree at the configured width

A matching tree's width came from its inputs: `matching_partitions` took
the selected source's count, or the maximum of every input's. In
standalone a cross-table statement arrives as one `MergeScanExec` per
table with one partition each, so `m / n` matched on a single worker while
the scans under it ran in parallel. The engine's
`execution.target_partitions` was reachable - `plan_fused_series` already
uses it for `series_scan` - but never reached this decision.

Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which
resolves a matching tree's width as `target_partitions.max(1)` for both
the same-source and the multi-input case. `to_execution_plan` keeps its
signature and its input-derived width, so a caller that has no configured
parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do
not consult the value at all: a per-series tree runs on whatever
partitions it was scanned with, and asking for a router there would only
add a shuffle.

Forward the value through `FusedSeriesExec` because `with_new_children`
rebuilds from the node's own children, which are the ones below the
router; rebuilding without it would resolve the width again from the
pre-router count and silently fall back to one.

Verified on a real standalone: the two-table segment now reports four
partitions with a `SeriesRepartitionExec` under each one-partition region
scan, where it reported one before, and the rows are identical.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): declare a keyed test frame's ordering, and order its keys

`plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`,
which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it
returns packs what `RawScript::ordering` reports, which is now the key-then-time
order this helper models. The node reads the ordering off the *plan* schema, so the
two disagreeing meant the router scheduled these fixtures as `Contiguous` while
unpacking validated them under the stricter ordered contract.

Build the schema through `frame_schema_with` with the ordering the stream actually
produces, so the plan and the runtime frames state the same contract.

That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with
both sides claiming `contiguous`, the ordered validation never ran. Running it
showed the matching fixture feeding key `b` before `a` within one partition, which
its declared ordering forbids. Reorder the partition; the two pieces of `host="a"`
still arrive in different partitions, which is what the test is about.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): declare the key order a two-phase raw scan actually keeps

`SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a
`TwoPhase` scan that is not reading columnar artifacts. The data that path
produces is ordered by primary key:

- candidates are merged and sorted ascending by the encoded primary-key
  column, then decoded to ids by walking that ordered stream without
  reordering;
- partition assignment is a deterministic TSID-to-partition map that
  appends in arrival order, so one series never spans two partitions;
- each assignment is drained completely before the next is taken, rather
  than emitted in completion order;
- the raw adapter's predicate buffering preserves input order and cannot
  let a later key overtake a held one, and two-phase records always report
  `ends_package: false`, so it never engages the package-tail machinery
  that legacy relies on.

Declaring the weaker order costs two things that are available: the frame
codec's `SequenceValidator` never ran its increasing-key check, and a
repartition owner could not use a key watermark to release an identity
before another input ended. It also contradicts the scanner's own
documented contract, which states the output is always ordered by
(primary key, time index) within a partition.

Report the order the path actually keeps, and reword the two ordering
errors that said "legacy" now that both modes can reach them.

Cover it with a test that forces two-phase mode and the non-prepared
fallback, asserts which of the two it ran, and then checks per-partition
key monotonicity, per-series timestamp monotonicity, single-partition
ownership, and the ordering the scan reports. The fixture writes three
rounds, uses two output partitions and a residual predicate, and keeps
one series above a batch so the multi-batch path is exercised.

The three existing two-phase raw tests asserted the old declaration. They
check that a series stays in one run and never recurs non-adjacently,
which the stronger order still guarantees; only their expected declaration
and their names said "contiguous", so those change and the substantive
assertions stay.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw match only by a key the computation keeps

`same_source` answers "this subtree reads one source", and a `MatchingLocal`
classification was taking that answer as licence to insert the router below
the fused computations, hashing the labels as they are at the raw stage.
Those are different facts. A math map drops the metric name and the
`__type__`/`__unit__` metadata, `timestamp` does the same, and every range
function drops them except `last_over_time` and `first_over_time` — so
`on(__name__)` over `abs(...)` routes by a key the very next node deletes.

Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side
routes `x` and `y` to different owners; both become the empty key after
`abs`, nothing re-partitions between the map and the match, and the pair is
silently lost.

Decide preservation separately from source identity. A node keeps the key
when it cannot remove one of the labels the key is built from; a `Window`
consults `RangeFunction::drops_metric_name` rather than assuming, so the
two functions that keep the name still route early. Anything not proven to
keep the key — a label rewrite, a histogram map, an unmodelled node —
does not establish preservation, and the tree is classified `SinglePartition`.

That is not a fallback that trades parallelism for correctness. When the
computation removes the labels the key is built from, every series on one
side reduces to the same empty key, so every left series matches every
right one: the match is whole-vector by construction and has no partitioned
form that preserves its result. One partition is the only placement that
answers it. Where the key does survive, the match stays partitioned.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): describe an admitted operand by what it produces

`try_plan_native_operand` handed the binary matcher the context saved
during scanning, which describes the operand's *input* layout. An admitted
call can change that layout: `histogram_quantile` folds a classic
histogram and drops `le`, so the scan context still names a tag column the
operand no longer produces, and default matching then builds a join key
over a column that is not there.

Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)`
declines whole-tree planning (two selectors over one table) and is then
admitted operand by operand; each operand drops `le`, and planning fails
building the join with `FieldNotFound { field: "le" }` against a schema of
`timestamp, field_0, host`.

Publish only the tag and field columns that the operand's own plan schema
actually contains, and drop `use_tsid`, which the produced output does not
carry. The check is against the schema the node returns rather than
against a list of functions that happen to shrink it, so a call added
later cannot reintroduce the gap by not being on a list.

Covered by a classic-histogram fixture with `host` and `le` and a float
bucket field, asserting each operand's context reports `["host"]` and that
the planned join key omits `le`. Reverting the change fails that test with
the `FieldNotFound` above.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the partitioned duplicate-group answer, with the gap named

`native_hybrid_binary_05_06_08` asserted that the right side's two
`job="other"` series are refused as a duplicate match group. On the
partitioned route they are not: the check is gated on `lhs.presence &&
rhs.presence` computed from the partition's own slices, so the partition
holding `other` sees no left-hand data and filters every point out.

Pin what the route actually answers, and state the three things that
constrain the fix, because each has already been tried and rejected here:

- the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a
  Prometheus-generated case where the left is absent at one point and the
  right carries three samples of one group there, and the answer is a
  value with no error;
- the scan cannot run unconditionally, for the same reason;
- the match cannot be collapsed to one partition to make both operands
  whole, because that gives up the parallelism this route exists for.

What remains is to let the verdict see whole-operand presence while the
partitions keep answering — a deferred check rather than a different
placement. This test now makes any change to the current answer visible
instead of silent.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a nested match instead of gathering it

A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` —
passes the logical admission check, because `independent_scan_pair`
recurses through `scan_operand`. The physical locality classification does
not go that far: it recognizes a match only when both operands read one
source each, so `same_source` fails on an operand that is itself a match
and the whole tree is classified `SinglePartition`. The fused computation
then ran on one partition while the scans under it stayed parallel, which
the existing planner does not do: it builds ordinary equijoins that
DataFusion can execute as partitioned hash joins.

Admission and placement are different questions, so they now get different
predicates. The tree is segmented when it is cross-table, not multi-input,
classified `SinglePartition`, and its root is a vector match with a nested
vector match directly under it. The inner match becomes a segment and the
outer one repartitions the evaluated operands by its own key, which is
what the segmenter already does for any exportable evaluated prefix; a
nested match whose two levels use different keys is routed twice, once per
key, rather than pretending one routing serves both.

The predicate stays narrow on purpose. Three-selector trees over distinct
metric logical ids are `multi_input` and keep their existing construction,
including the parent binding that construction produces. A same-table
nested match does not enter this branch either, so its prior admission is
unchanged.

The plan test's collector walks each fused node's native tree in
expression-child order and resolves each input's `source_index` against
that node's inputs, so a segmented plan is checked for the same property
as before — every selector occurrence bound exactly once, in expression
order — rather than being checked more weakly. It panics on a shape it
does not model instead of returning an empty list.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): prove a raw match key survives the computation

`preserves_matching_key` decides whether a subtree keeps the labels a
vector match routes by. Two arms answered yes without looking at the
field that decides it.

A binary with a scalar operand recursed into the vector side alone, but
arithmetic and a `bool` comparison drop the metric name while a filtering
comparison keeps it — the distinction execution already makes. And an
`Instant` recursed unconditionally, while `timestamp(x)` compiles to
`InstantValue::Timestamp`, which drops the name too; the check written for
the separate `Timestamp` node never covered that shape.

Both now read the field that decides: the operator together with
`return_bool`, and `InstantValue`. The labels checked are the whole
computed set — name, type and unit — because `drop_metric_name` removes
all three and default matching does not ignore the last two by itself.

Without this, `(x * 2) + on(__name__) (y * 2)` was classified
matching-local, the router hashed the raw names before the maps dropped
them, and a valid pair could land in different partitions with nothing
re-partitioning afterwards. The regression asserts the value is 10 and
that the tree is `SinglePartition`, computing the destinations the router
would have used (`x0` to partition 0, `y0` to partition 1 of two). With
the old arm restored the value assertion fails with `left: [10.0],
right: []`.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): find a nested match through the wrappers around it

Segmentation was gated on the root itself being a vector match with a
nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never
entered: the wrapper or the scalar broadcast sat where the match was
expected. Meanwhile the segmenter already walks children and leaves a
wrapper in the parent while cutting what it can carry, so no per-wrapper
case is needed — the search just has to look through them.

Replacing the root check with `carries_prefix` was too wide: almost any
multi-source tree contains something a boundary could carry, so ordinary
shapes like `some_metric unless on(tag_0) other_metric` were cut for no
gain and their inputs stopped being raw sources.

So keep requiring a nested vector match, and search for it anywhere in the
tree rather than at the root. Wrappers and scalar broadcasts are
transparent because the walk visits every child; the nested match itself
is still required, which is what keeps the plain multi-source shapes on
the plan they had.

`cross_table`, `!multi_input` and the `SinglePartition` locality
requirement stay, as before: they are what keeps distinct metric logical
ids and the same-table shapes out of this branch.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): let a router publish an identity whose chain is complete

`SeriesRepartitionExec` published an identity only once every input had
contributed it or ended, because an identity may be split across inputs. A
per-series region scan assigns each identity to one partition, so on that
shape the rule holds every identity until the other partitions reach EOF: a
region's whole raw selection is retained before the first series leaves.

The node now carries an `IdentityCompletion`, defaulting to the existing
merge. `CompleteDisjoint` publishes an identity as soon as its chain closes
and keeps the published identities charged in the pool, so a second chain
for the same identity fails the execution instead of emitting a partial
series as final. The declaration is the producer's, not something a consumer
infers from a node's type: publishing early against a producer that does not
own its identities would be wrong, so the conservative mode stays the
default and every existing constructor keeps it.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): choose a series-frame root's router from declared provenance

`SeriesFrameOutput` normalized every multi-partition frame root through a
router that merges identities, which is wrong for a root whose identities are
already disjoint: it waits for every scan partition to end before the first
series leaves.

The rule now selects `CompleteDisjoint` only for a root whose producer
declared that property. A raw root qualifies when `RawSeriesSourceExec` was
built through `to_per_series_execution_plan` - which also checks the child is
a `SeriesScanExec`, so the type alone is not taken as proof - and a direct
`SeriesRepartitionExec` root qualifies because it merged the identities
itself. Everything else, including a fused frame root, keeps the merging
router: a native label transformation can map two source series onto one
output label set, so its output identities are not exclusive.

The rejection of a key-only raw frame is unchanged and its reason is
restated: coalescing cannot produce such a frame, and a router needs explicit
labels to check that one key has one binding.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): report the labels `absent` names, not the table's columns

`fused_output_tags` answered `None` for `absent`, so the caller fell back to
the existing planner's column names. Those are the table's tag columns, which
is not what `absent` reports: `absent(m{host="x"})` carries the labels its
matchers name, and the compiled tree already knows them in
`BoundVectorOp::Absent { labels }`.

The label set is now read from that field: the named labels are kept in the
input's column order, so `absent(native_shapes{host="missing"})` over a table
with `host` and `le` reports `host` rather than `ts, val, host, le`. A label
the matchers name that is not an input column still answers `None`, because a
frame has no column to carry it; `count_values` and `histogram` are unchanged.

Caught by the native-versus-legacy value comparison: the extra column made
`absent` differ from the existing plan while every value matched.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): describe a frame root's emission from what it does

`FusedSeriesExec` declared `EmissionType::Final` for a frame output, but
`poll_output` hands a frame's evaluated chunk over while its input is still
arriving, so the declaration overstated the node to the optimizer. Only a
partial aggregate's state is emitted after EOF; every other output keeps its
input's emission type.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): compare the shapes the native route now admits with the existing plan

Removing the shapes a name list used to refuse put single-selector `sort` and
`sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route.
Nothing compared their answers with the existing plan, so the widening was
unverified.

The new case runs each shape both ways over one metric table. `sort`,
`sort_desc` and both `absent` cases compare the whole printed result, which
covers column names and values together. `histogram_quantile` compares values
row by row and asserts the one known difference instead of hiding it: a
segmented statement names its value column from the compiled tree (`val`)
where the existing plan names it after the expression that produced it. That
case is given a window long enough to produce a row, so it cannot pass by
comparing two empty results.

The `absent` cases caught a real difference: the native route reported the
table's tag columns rather than the labels the matchers name, which is fixed
in the preceding commit.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): publish a fused root's identities early when the tree keeps them

The root rule publishes an identity as soon as its chain closes only for a
root whose producer declared that each identity lives in one partition, and
it knew two such producers: a per-series raw scan, and a router that had
already merged its identities. A fused root was always treated as needing a
merge, so a region whose segment is a plain instant selection over one
per-series scan still held every identity until the other scan partitions
reached EOF - `sum(topk(1, m))` is that shape, and its segment is already
multi-partition with no label rewrite.

`FusedSeriesExec` now answers whether its output identities are disjoint, and
the rule asks it. The answer is narrow on purpose: one input, and a tree of
nothing but inputs and sample instant selections, over a per-series scan or
another fused node that already answered yes. Anything that computes labels
is excluded, because a timestamp or a pointwise map can map two source series
onto one output identity, and publishing early would then hand over one
identity's pieces as if they were different ones. A vector-to-vector binary
reads across series and is excluded too.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): segment a computed matching key's operands instead of gathering them

A vector match whose operands compute away one of the matched labels was
classified `SinglePartition` and executed on one partition. That is correct
for the bare tree - the raw names cannot route a match whose operands drop
them - but it also gave up the parallelism the operands still have:
`(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so
the computed sides can be routed by what they actually carry.

The planner now cuts those operands into segments of their own and lets the
parent route their evaluated outputs by the unchanged match key. The key
itself is not rewritten: `group_key` reads the labels present at that point,
so `On(__name__, host)` degrades to the `host` a computed operand still has.

Which shapes take this route is decided by one predicate shared with
`input_locality`, so the admission rule cannot drift from the locality
judgement: a matching site whose two operands are each proven single-source
computations, at least one of which does not preserve the key. A failed
preservation proof on its own is not enough - it can also mean the proof gave
up, as with a parameter that is a compound expression or an operand that is
itself a vector match - and the shared doc says so.

The same table read twice is admitted for this shape only. The rule that
declines several selectors on one table exists to keep folds such as
`sum(m) / m` off this route, not because one table cannot be read twice; the
compiler already gives each occurrence its own source index, and each cut
gets its own input. Distinct partitions are what the two operands need, and
they get them.

The regression executes both the same-table and the cross-table shape over a
four-host fixture and asserts that at least two output partitions produce
rows, so the result is not silently produced by one of them.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): take the early-publication guarantee from the scan's binding

The root rule publishes an identity as soon as its chain closes only when the
producer can promise that each identity lives in one partition, and the fused
answer to that question asked whether its input was a `SeriesScanExec`. The
node's type is not that promise: a scan binds a set of logical tags and decodes
those into the series' labels, and `try_to_named_series_scan` requires the
caller to pass complete tag names but does not check it. A binding that names
only some primary-key columns maps two stored series onto one output identity,
and early publication would hand one identity's pieces over as different ones.

The answer now reads the binding: the label source has to be `PrimaryKey`, and
every primary-key column has to be among the bound tags. A source that carries
its labels explicitly is not this shape and answers no, as before.

Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is
false, full coverage is true, an explicit source is false - and by tests for the
predicate's two other parts: a tree that computes labels and a tree that is not
one instant selection over one input both answer no, and a gated execution
shows a completed identity leaving a fused node while the other input is still
held.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): drop the checks the types and decoders already guarantee

Two re-checks ran inside the repartition node over facts its inputs had
already established, and both cost real work on every query.

A raw merge scanned each chunk's timestamps for strict increase, but
`RawTimestamps` cannot hold an unordered buffer: its field is private, its only
constructor validates, and the accessors are read-only or order-preserving, so
no safe caller can build the value this check defends against. The check is
gone; the identity, label and shape checks around it stay, because those
compare different producers' pieces rather than one chunk's own timestamps.

An owner rebuilt each chunk's identity to prove a chain does not change it, but
the decoder that hands the chain over already refuses that: `SequenceValidator`
holds one open key and rejects a key or metric change before the chain closes,
`EvaluatedSequenceValidator` does the same for labels, and the input task that
assembles chains only reads from those decoders. What stays is the identity the
owner needs anyway to pick a destination, now built by the named functions
rather than inline.

Where a schema was needed, the node built an empty stream of its own, wrapped it
in a packer and read the packer's schema back. The frame module already answers
that question from the declaration, so the fake stream is deleted along with its
half of the roundtrip.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recursively segment native matching plans

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): derive native schemas without legacy probes

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recognize complete metric scan identities

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23)

* fix(promql): release completed disjoint identity history

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c)

* test(promql): assert native aggregate schemas independently

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): identify internal metric keys by reserved id

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep parallel sources and derive group match labels

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe the frontend fused root in scan-parallelism checks

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): release duplicate evidence immediately on abort

Signed-off-by: discord9 <discord9@163.com>

* test(promql): preserve operand identity across matching repartition

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): charge the final duplicate verdict until it is reported

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe native scans across frontend and region plans

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): drop dead test helper and clippy lints

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): select limitk by canonical label order

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep data-free subtrees in the parent segment

Signed-off-by: discord9 <discord9@163.com>

* test(promql): prove the duplicate error crosses matching partitions

Signed-off-by: discord9 <discord9@163.com>

* test(promql): drop debug prints from the segmentation test

Signed-off-by: discord9 <discord9@163.com>

* revert(promql): keep limitk's Prometheus input-order semantics

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): satisfy clippy in fused root schema

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-04 14:12:56 +08:00
fys b6cf3384dc fix: register trigger procedure loaders through DdlManager (#10123)
Add a loader registration hook to TriggerDdlManager and invoke it
  before procedure recovery. Test that registration reaches the
  configured trigger DDL manager.

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-04 04:31:23 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Weny Xu 7339a56abd feat: share streaming range results across scans (#9794)
* feat: share streaming range results across scans

Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.

Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): clean up shared range registrations on release

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 11:33:34 +00:00
Weny Xu 6491700fe6 fix(meta): forward flushed Parquet bytes in region statistics (#10100)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 11:42:56 +00:00
Weny Xu 295bed98ec feat: share Parquet byte page loads (#9793)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 10:52:31 +00:00
shuiyisong a57ab18eaa feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries

Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers.

Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(servers): coalesce flow notifications

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(ingest): reduce write progress overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): drain confirmed writes on cancellation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(batcher): reduce row owner allocations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(batcher): restore per-flush metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* revert: scope retry-hint fix to storage writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): restore flush metrics and benchmark

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): propagate sync batch write errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve peer batching for confirmed writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: avoid unnecessary row owner copies

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: reuse IPC buffers in tracked retries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* style: order write dispatcher imports

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-02 03:09:13 +00:00
Weny Xu c035670ec0 feat: share Puffin metadata and index page loads (#9792)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-30 07:25:55 +00:00
zhaiAohan 8a38b90135 feat: add relational region engine skeleton and registration (#9953)
* feat: add relational region engine skeleton and registration

Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered.

Signed-off-by: tian1220A <1573612521@qq.com>

* feat: add relational connection and idempotent region lifecycle

Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder.

Signed-off-by: tian1220A <1573612521@qq.com>

* refactor: address review comments on relational region engine

- remove connection.rs (only used by its own tests); it will be
  reintroduced when connection handling is wired up
- drop RelationalOptions::version (not validated or used to select a
  deserialization format)
- validate an idempotent create against the existing region definition
  (region id / schema / primary key / options) and return Unexpected on
  mismatch, with a reason field on RegionExists
- use serde.workspace and remove the empty [features] section
- cargo fmt

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: adapt relational engine to latest main

- implement RegionEngine::set_region_role_with_lease (no-op lease, the
  engine takes no writes so no lease is needed)
- pass node_id to build_object_store_manager in the registration test
- drop stale url dependency from the relational-engine lock entry

Signed-off-by: tian1220A <1573612521@qq.com>

* ci: register relational-engine in scope rules

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: remove table self-dependency from Cargo.lock

Signed-off-by: tian1220A <1573612521@qq.com>

---------

Signed-off-by: tian1220A <1573612521@qq.com>
2026-10-30 04:06:01 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00