Commit Graph
977 Commits
Author SHA1 Message Date
discord9 b30557960b feat(promql): preserve scan parallelism through native frame boundaries (#10055)
* fix(promql): let a closed chain keep its charge until the packer is done

Four places released the frame reservation: the early-publish path handed it to
the packer, but the queue drain and the end-of-input path freed it in place. Those
two run while the packer still holds rows it has not turned into batches, so a
series spanning several batches went unbilled once its last piece was handed over.

Every emit path now hands the charge over, and the packed stream frees what it
holds: what is left when it is polled and pending is the one identity it is
turning into batches, and nothing survives its end or its drop.

The identity guard is also widened: within one source every series shares its
metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is
what a range function, a math map and the timestamp shape do -- still maps
distinct series to distinct labels. Only rewriting an ordinary label can make two
equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame
output therefore publish as their chains close instead of buffering a partition.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must

Three defects in how this node decided a tree's execution layout.

A match whose two operands derive from the same source still needs its match keys
co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)`
therefore fell through to single-partition execution and lost the parallelism the
scan gave it.

A raw input never received a matching router: the normalization branch returned it
unchanged unless the whole tree was global, while the builder requires a direct
matching router for any multi-partition matching-local input. A two-source raw
query was rejected at planning.

Nothing reconciled the partition counts of different sources either, though the
builder requires one common positive count. A single count is now computed for the
matching-local case and every router under it uses it, gathering when that count is
one.

The window restriction also ran before normalization decided the layout, so a tree
about to be gathered could be refused for a limit that only applies to partitioned
windows.

Two tests changed with the behaviour rather than around it: hash routing does not
promise which partition holds a given series, so that test compares results
instead of placement while keeping its partition count, per-partition execution
and operand-order assertions; and a test that pinned the old refusal of a
multi-source frame now pins its acceptance.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): keep a raw source's scan partitions instead of gathering them

`RawSeriesSourceExec` declared one partition and read every child partition
inside `execute(0)`, so a cross-table node lost the scan's parallelism before
its inputs reached it. It now declares the child's own count, executes exactly
the partition it is asked for, and re-labels that partition's frames.

That leaves `SeriesGather` with no consumer: it existed to interleave several
partitions into one stream. The module is gone rather than kept as a hidden
N-to-one path a later plan could quietly fall back on.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): publish a raw source's frames under the ordering it declared

`pack_raw_series` derives the frame schema from the stream's own ordering, and
`RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing
the scan's stream through unchanged broke that agreement: a columnar scan
promises key-then-time, so the region published key-then-time frames while the
frontend still expected the schema it had built, and `MergeScan` refused them.

The forwarding adapter restates the ordering this node declares. It is the
weaker promise, so a scan that keeps a finer one still satisfies it, and every
other stream property is forwarded unchanged.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw frame that names its labels by its key

A region cannot hand out complete labels: it has no metric-name binding, so
`prepare_raw_scan` refuses an explicit binding outright. What it hands out is
the stored key plus the description a decoder needs, which is the normal input
to this node rather than a case to refuse — every single-table selector is
planned that way. The node's constructor refused exactly those frames, so any
plan that inserted a router under a single-table selector failed before it ran:
the matching policies this branch adds, and the one-partition normalization of
an evaluated tree.

The node publishes labels, because that is what it groups and closes identities
by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the
input is decoded once, at the unpack boundary this node already has, and an
already-labelled stream passes through it untouched. The frame it declares is
rebuilt from the input's own declaration instead of being compared against it,
which keeps the check that a frame this node cannot re-declare is refused while
letting input and output differ in how they name the same series. `frame_schema_with`
becomes public for that rebuild, since it is the one place that decides what a
declaration is.

`FusedSeries` now describes the router's output rather than the scan's when it
wraps a raw input, because the binding it hands `build` has to match the frame
the router actually publishes.

Tested through the node directly: a key-labelled input routes at all; matching
puts both pieces of one label set in one owner and closes it once; and
`RawPrimaryKey` still hashes the key rather than the decoded labels.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): measure a routed pair by the width it routes to

Four assertions still described the layout this branch replaced: a node with two
inputs used to read every partition of both into one and answer one partition,
and that gather is gone. The pair is now matched in the partitions it was
scanned in, so the fused node answers as many partitions as its inputs route to.

Each of these fixtures scans with four partitions, and on the base of this branch
all five tests passed, so this is a behaviour change this branch made and did not
re-verify rather than a defect: the value assertions behind these shape
assertions pass unchanged, in both the native and the legacy comparison.

The width is read from the routed children instead of being written as four, and
the scan is found by walking down through the nodes that only pass its frames
along, since a routed input is no longer a bare `SeriesScanExec`.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): normalize a multi-partition series-frame root

A query engine hands its plan to one stream, and for a plan with several
partitions that means `CoalescePartitionsExec` - ordinary batch coalescing.
A series frame cannot survive it: one stored series may be split across
partitions, and a chunk's `is_last` closes that producer rather than the
series, so interleaving two partitions' batches produces a stream whose
decoder refuses it (`raw series key changed before final chunk`). Evaluated
frames have the same shape of problem: their validator keeps one open chain,
and a batch from another partition interrupts it.

The engine's root is also a region's root, which is how a cross-table native
statement reaches this: its per-region sub-plan is rooted at a partitioned
`RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end,
before the final sanity check, and it normalizes the root and nothing else -
`FusedSeries` already inserts its own owner wherever a consumer needs one.

A single-partition root is left alone: one producer is the whole answer. A
key-labelled raw root is refused rather than coalesced, because the router
checks that one stored key decodes to one label set and it needs the labels to
do that. No production plan reaches that rejection today - the only producer of
a key-labelled frame is the region's series scan, and every plan that executes
it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish
explicit labels - and the rejection is what keeps it that way.

`schema_check` stays enabled: for the roots this rule accepts, the router
re-declares its input's frame with the same content, label mode, decode
description and ordering, so the schema it publishes is the one it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): admit an evaluated frame as a fused input

The extension planner hands a child straight to the fused executor when that
child already publishes a series frame, and until now it only recognised a raw
one. A segment's output is an evaluated frame, so a statement planned as two
segments could not be bound above the first one.

The check it replaces asked for a raw frame; the property that matters is that
the frame names its own series, which is what `labels() == Explicit` says -
evaluated frames always declare it and carry no decode description. Everything
else stays with `FusedSeriesExec::build`, which already refuses a mixed-content
node, an evaluated input that is not contiguous, and a frame whose declaration
does not match the binding it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): plan a global topk under an aggregate as two segments

`sum(topk(1, m))` needs the whole vector for its ranking, which is a property of
the tree rather than of the statement, and the routes that existed could only
answer a whole-vector tree in one partition: the statement fell back to the
existing planner. Splitting it lets the per-series prefix keep the scan's
partitions while the ranking sees every candidate.

The cut goes below the `topk`, not below the `sum`. A segment holding a `topk`
reads across series, so it would gather its own input before evaluating the
prefix - correct numbers, no parallelism, which is the thing the split is for.
Below the `topk`, the prefix runs on the scan's partitions and the ranking and
the fold above it run once, on the child's evaluated frame.

Both segments are built from one compiled tree: the child is the prefix subtree,
the parent is the original tree with that subtree replaced by an input bound to
the child's frame. Nothing is re-parsed, so the grid, the matching rules and the
bound scans are the ones the compiler produced.

The parent needs its output columns without asking the existing planner to name
a computed column after its expression, so `fused_output_tags` now derives the
grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels
through. It derives output *columns*, not a complete runtime label set: the
metric name is bound separately from the tag columns, and a grouping by
`__name__` keeps that name in the labels while contributing no column.
`fused_root_schema` is that derivation, shared with the ordinary route.

The segmenter is its own method, boxed at the call site. Inlined, its future
made every frame of the planner's recursion large enough to overflow the stack
on a deeply nested statement - `test_parse_or_operator` aborted before this.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the two-segment topk plan and what it keeps

The statement's two segments do not land in one physical tree: the prefix has
one scan for a child, so the categorizer classifies it commutative and pushes it
into the region, while the aggregate above it stays at the frontend. The
observer therefore records every plan a query engine wraps, not only the first,
so the test can look at the region's own root.

What the assertions pin, and why each one is worth its brittleness:

- the answer is the global maximum, not a sum of per-partition winners;
- the frontend segment holds the complete sum and the global topk over an
  evaluated input, with no instant evaluation left in it;
- the region's segment holds the prefix and its output is an evaluated,
  explicitly labelled, contiguous frame;
- that segment keeps more than one output partition, which is the whole point of
  cutting below the topk;
- the region's root is the series-aware router, one output, and it reads that
  segment.

Column names are not compared: this route names a computed column after its
compiled root while the existing planner names it after the expression, which is
older than this split and not what this test is about.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): share the input-locality classification

A tree's distribution requirement is one property, and two layers now ask it.
The physical planner already decided with `input_locality` what a tree needs
from its inputs; the logical planner has to ask the same question when it splits
a statement, or the two can disagree about what a subtree requires.

`InputLocality` and `input_locality` are therefore public within the workspace.
The classification describes the external inputs a tree needs - not its output
distribution, and not which neighbouring operators may share a node - so a
caller that reads it as either of those will cut where it should not.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement where a gather would swallow its prefixes

The previous commit planned `sum(topk(k, prefix))` by matching that shape. A
planner that grows one matcher per query shape grows one per benchmark entry,
and the shape is not what decides where a statement divides: the tree already
says what each subtree needs from its inputs, and `input_locality` answers it.

So a segment is now a maximal subtree with one distribution requirement. Inside
a tree that must gather, any child that (a) can cross a boundary - it produces
evaluated samples rather than raw samples, aggregate state or a scalar - and
(b) does not itself need a gather, becomes a segment of its own and keeps the
partitions it was scanned with. Nothing else is cut, and a tree with nothing to
cut keeps the plan it had.

Every segment rebinds its own inputs: the scans it reads directly first, in the
order it first names them, then the segments cut out of it, because
`FusedSeries` requires dense source indices over its children. `fused_output_tags`
derives the parent's columns from those inputs rather than from the statement's
first selector.

Three gates keep this from answering what the routes before it would not. Each
one is a failure this change produced before it was added:

- the existing planner must still be able to plan the statement. Deriving tags
  locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` -
  an empty range, which the existing planner rejects - was answered instead of
  failing;
- every segment's tree must construct a `NativeExecutionState`, because nothing
  below can fall back once physical planning starts, and a grouping by a value
  field is a shape the runtime refuses;
- the tree must contain a prefix that a boundary could carry, so a whole-vector
  tree with no per-series work inside it is not split for nothing.

The scope is one selector. A statement over two tables binds each scan as a
`RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in
a region, so its prefix would not gain the partitions the cut is for; those
statements keep the plan they had, which the acceptance test pins.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the segment boundaries and their scope

One statement per shape the cut can meet, asserting what the rule says rather
than what one query happens to produce: a per-series tree stays one node, a
matching tree routes instead of gathering, a global fold over per-series work
cuts that work out, a constant stays embedded because a boundary cannot carry
one, and a statement over two tables keeps the plan it had because a segment
over `RawSeriesSource` cannot be placed in a region.

Answers are compared as sets of rows with the value column renamed. Three
things differ between the routes and none of them is the answer: this one names
a value column after its compiled root while the existing planner names it after
the expression, the two emit columns in a different order, and this one is
partitioned so its rows arrive by partition. Comparing them as what they are
leaves the values as the only thing an assertion can fail on.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep a selected segment whole

A segment is chosen because its subtree needs one distribution; cutting inside
it again contradicts the choice. The inner builder was still applying the cut
predicate, and that predicate is "exportable and not `SinglePartition`" - which
every node inside a per-series subtree satisfies, since a per-series tree has no
`SinglePartition` node in it to begin with.

`sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one
segment, then `abs` was cut out of that segment, leaving a middle segment whose
child is a node rather than a scan. That middle segment cannot be placed in a
region, so it answered one partition while the scan had several - the prefix
lost exactly the parallelism the cut is for.

The inner builder now collects the segment's `Input` leaves and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): bind a segment to its own scan

A cut segment carries one source's identity: its scan, its metric name, its
time index, and its labels. Building it from the statement's first selector
only happens to be the same thing while a statement has one selector.

`build_inner_segment` takes the two lists the planner already has - each
occurrence's filtered scan, and each occurrence's metric, time index and tags -
and binds the segment's single source from them. The parent keeps the wrapped
children and the statement-level binding it has now, because a source that
reaches the parent directly still has to cross a region as a raw frame.

A segment that collects more than one source is declined. `MatchingLocal`
satisfies the cut predicate while reading two sources, and one segment cannot
carry both sources' identity.

No behaviour changes: the entry is still one selector, so no scan is wrapped
and the statement-level values are already that selector's.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement that reads more than one table

A whole-vector statement over two sources is now cut into one per-series
prefix segment per source, with the fold above them in the frontend. Each
segment binds its own source's scan, metric, time index and labels, so it can
be placed in the region that holds that source, and each prefix keeps the scan
partitions it was read with.

Three things had to change for the existing segmenter to reach that shape:

The cut sat inside `!per_partition && !multi_input`, so a statement over two
metric tables - which is `multi_input` - never reached it. The cut now sits
under `!per_partition` alone. The aggregate split keeps its own conditions,
including `!multi_input`, so a multi-input tree is still not split that way.

The single-selector gate was doing two jobs. A statement naming several
selectors declines unless those selectors read more than one table: the count
alone says nothing about whether the tree has more than one input to place, and
the shapes it would otherwise admit - one metric named twice under a fold, such
as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers.
`cross_table` rather than `multi_input` decides this, because two Mito tables
carry no metric logical id and are still two sources.

A segment that collects more than one source is declined. A matching subtree
over two sources is a cut candidate, and one segment cannot carry both sources'
identity, so `sum(topk(1, m / n))` keeps the plan it had.

The segmented parent reports the value column the compiled tree names, not the
one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where
the existing plan reports an expression-shaped name. The values are identical
and both names are pinned in the test, so a change to either rule shows up
there. Restoring the existing name needs a rename above the parent - the node's
own schema must keep the compiled names its output columns are classified by -
and is left to a follow-up.

Signed-off-by: discord9 <discord9@163.com>

* test(query): pin that two sources keep their own scan bounds

A segmented statement binds each prefix to its own source's scan, and A1 made
that binding: `bare_scans` is derived from the prepared scans after each
selector's time filter has been pushed into its own `TableScan`. Nothing tested
that the two segments actually read `their` bounds rather than one window applied
twice - the existing two-source case has no offset, so both sides read the same
range and a crossed binding would answer the same numbers.

`sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only
in the right side's window, so the two queries must disagree: 20/1 and 30/2
against 20/2 and 30/3. The test asserts they do, which is what fails if both
segments were given the left scan's bounds.

The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints
only the peers: the debug form carries the table each segment reads and the
embedded `FusedSeries` segment. An in-process region hands the plan observer no
separately wrapped region tree, so the per-partition evidence for segments is
measured in `native_aggregate_test.rs::check_two_source_segregation` instead.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): decline a multi-source tree that groups by the metric name

The gate that lets a multi-source statement reach the segmenter is
`selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above
it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`.
Two ordinary Mito tables differ in table name and carry no metric logical id, so
they are `cross_table && !multi_input`: the guard did not apply and the statement
reached the segmenter.

`fused_output_tags` keeps a grouping by `__name__` out of the declared columns on
purpose - the name travels in the chunk's labels, not as a tag column - so
`sum by (__name__)` over two sources derives no tag column while the runtime
holds two groups that differ only by metric name. Both are then reported under
one label set, as two rows a reader cannot tell apart. That is the failure the
metric-logical-table test already documents; it could not catch this path because
its tables make `multi_input` true.

The check is scoped to a tree that reads more than one source. One source folds
one metric, so its groups carry the same name and naming that name or dropping it
cannot merge two of them.

Signed-off-by: discord9 <discord9@163.com>

* test(query): cut a range function on each side of a division

The two-source cases so far cut an `Instant` prefix: one sample per grid point.
A window function's prefix computes its window inside the region over that
source's own samples instead, so the segment carries a `Window` rather than an
`Instant` and nothing tested that such a prefix is still a cut candidate.

`sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step
the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then
4/3. Two different step values are what let a shared window bound change an
answer, which is asserted before the result is compared.

The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset
case: an in-process region hands the plan observer no separately wrapped region
tree, so the per-partition evidence lives in `native_aggregate_test.rs`.

This is the shape of the throttling query in the benchmark corpus: a window under
a fold, on each side of a division.

Signed-off-by: discord9 <discord9@163.com>

* test(query): plan the benchmark's throttling and namespace-ratio shapes

Every two-source case so far divides two ungrouped scalars. The corpus queries
that motivate this work group by a label on each side, so each segment reports
several groups and the parent matches the two sides by that label. Nothing
tested that path.

`container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total`
is the corpus query verbatim: two metric logical tables on one physical table,
`sum(increase(...)) by (container)` on each side. It plans as one frontend
`FusedSeries` over two `MergeScan` children, each holding a segment bound to its
own metric, and reports the same labelled rows as the existing plan.

The namespace-CPU-usage-over-requests shape differs from the corpus text only in
the metric names, because the corpus names carry recording-rule colons that a
table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form
and is asserted as that shape rather than as the query verbatim.

Both tests assert the native route answered, so a decline fails here instead of
passing on the legacy rows. The value column still differs from the existing
plan's - the same known gap recorded for the ungrouped case - and both names are
pinned.

The fixture helper records the three setup traps the hard way: `INSERT` needs
explicit column lists because the physical table's column order differs from the
logical table's, SQL keywords used as tags need quoting, and no sample may sit at
timestamp zero, which both the scan's lower bound and PromQL staleness drop.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record which native route every corpus query takes

The corpus is 29 queries and nothing said which of them the native route
answers, or by which of its plans. One parameterised test now runs all of them
over generated fixtures and records, per query, the route and the numbers.

Four routes, and the frontend tree alone cannot tell them apart:

- `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one
  per source; the prefix work happens in the regions.
- `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region
  `FusedSeries` that reports `partial state:`, so what crosses the boundary is
  aggregate state.
- `region-local` - the frontend holds no native node at all: a per-series tree is
  commutative, so the whole `FusedSeries` is pushed into the region. A classifier
  that only looks at the frontend calls these legacy, which is what happened
  before this test existed.
- `legacy` - no native node on either side.

All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two
`region-local`, none legacy. The two `region-local` ones are the corpus queries
that fold nothing - a bare selector and a `rate` - which is also why pushing the
whole tree into the region costs nothing.

The test asserts only that the existing plan answers each query with rows, and
that a native route's labelled rows agree to within `1e-10` relative. It records
the route rather than asserting it, because which route a shape takes is a
property of the planner that this test exists to report, not to legislate.

Ten queries differ from the existing plan in the last bits; the largest relative
difference is 2.9e-16, which is summation order, not arithmetic. The table prints
that figure per query so a future equality failure can be told apart from
rounding without re-measuring.

Every fixture here is generated: explicit `INSERT` column lists, quoted SQL
keywords, and no sample at timestamp zero. They establish the plans and the
values; they say nothing about speed, memory, or how many partitions actually
produced rows.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record how many partitions each corpus query's nodes used

The route matrix said which plan answers each query. It did not say whether that
plan produced anything, so a node that ran and emitted nothing looked the same as
one that worked.

`output_values_per_partition` reads `native.root.output_values` off a
`FusedSeriesExec` and returns the counts per partition, and the matrix prints them
as `partitions used`. `assert_four_native_partitions` keeps its own assertions and
now builds on the same helper.

What the column does not establish, and says so where it is printed: these
fixtures hold a handful of series each, so a scan puts them all in one partition -
every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the
plan. The one place a fixture is built to fill the scan's partitions is
`native_aggregate_test.rs::check_two_source_segregation`, which asserts all four
produce values for both segments of a two-source statement. This matrix records a
single assertion instead: at least one partition produced values, so a node that
ran and emitted nothing fails.

Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise
divisions. Their region side has no `FusedSeriesExec` at all, because both scans
are matched inside one region: that is the route, not a missing measurement, and
the table says `no metric` rather than implying zero.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record the new rule in the explain stages

`TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering
`SeriesFrameOutput` adds one line to that listing whether or not the rule changes
the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary
tables, whose schemas carry no series frame declaration, so the rule returns the
plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the
existing lines say for the same reason.

Regenerated by the sqlness runner; the diff is those three lines and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* test(query): make the corpus matrix prove what it claims

Three gaps, all in the test the "every corpus query answers natively" claim rests
on.

The partition helper returned early on a plan that reported no
`native.root.output_values`, so a producer that registered nothing passed on the
declared partition count alone. It now expects the metric: the point of the
assertion is that every partition executed, and an absent counter says nothing
about that.

`segmented` was decided by the frontend tree containing a `MergeScan`, which also
matches a frontend native node consuming raw frames from its regions - a
different placement that says nothing about a prefix running in a region. The
merge scan's `Debug` carries the plan its region runs, so the check now reads
that for a segment; the raw case gets its own `frontend-raw` route. Four queries
move there: L05, L06, L08 and R03.

The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been
probed in a throwaway run. They are entries now, with the join labels Q08 and Q09
match on and real `le` buckets for Q10's histogram. A fallback is no longer a
route to record: a case that reaches the existing plan fails, because that is the
regression this table exists to catch.

Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): let the compiler decide which calls this route answers

A second list of function names stood beside the compiler's own capability
check, and the two disagreed. `admitted_selector` named five calls; the compiler
folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and
more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same
call meant two different things depending on what wrapped it.

The compiler already answers this question: `compile_call` refuses a name it has
no kernel for with `NotImplemented`, and the planner turns that into a decline.
With the list gone there is one place that decides, and a call the compiler folds
reaches the route whether or not it is wrapped in an aggregate.

`sort` and `absent` read across a vector rather than one series at a time, which
is why the old list named them out. They compile, so they are planned; the tests
that asserted the refusal now assert the admission. Only the shapes the compiler
genuinely cannot express keep the existing plan - a subquery evaluates its body on
a grid of its own, which this route does not offer.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): admit an operand by its shape, not by call name

An operand of a hybrid binary was admitted only if it was a selector under one of
five named calls. The compiler folds many more than those five, so `ceil(m)`,
`irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under
an aggregate was admitted - the name list disagreed with the compiler about what
this route can answer.

What the list was really protecting is a layout assumption, and the shape states
it directly: `try_plan_native_operand` hands back a selector's planner context,
and the hybrid caller reads field columns and tags off that context, so an
operand whose output layout is not its selector's cannot be one. `sum(A) / B` is
refused for that reason, not because `sum` is a name.

`single_selector_spine` states it on the expression: a selector, parentheses,
unary, a call carrying exactly one vector-or-matrix argument, or a binary whose
other side is a compile-time constant. Calls are followed by the type of their
arguments rather than by `constant_value`, because a call's string parameters are
constants that `constant_value` does not fold and would otherwise look like data.

Which calls this route answers is now the compiler's answer alone: a name it has
no kernel for comes back as `NotImplemented`, which the planner turns into a
decline.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): mix raw and evaluated inputs in one fused tree

`build` refused a tree whose children were not all raw or all evaluated,
but nothing below that point assumed uniformity: `execute` picks the
decoder from each source's own schema, `FusedFeed` carries the source
index, and `collect_source_schemas` already validates a repeated source
against its own grid. A frame stream has one format per stream, not one
per consumer, so one segment can decode the raw samples of a rate and
compare the result against a vector a previous stage already evaluated.

Remove the refusal and make the remaining per-input decisions per source:
`evaluated_global` now asks about the source it orders, and
`window_sources` keeps only the evaluated sources, since a raw source is
answered by the raw path it always used.

Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y`
arrives evaluated on the statement grid. The test asserts all ten
(host, timestamp) rows and fails on the removed refusal if the refusal is
restored.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): bind a segment to every source its subtree reads

`cut_as_segment` accepts any exportable subtree that is not
`SinglePartition`, so a matching subtree over two sources - `m / n` - is
a legal cut point. `build_inner_segment` then required exactly one source
and returned `None` for it, and `segment_tree` propagates that `None` as
"the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back
to the legacy path even though its cut was legal.

Build the segment from every source the subtree reads. Each child gets the
`RawSeriesSource` boundary already used for separate metric tables, which
carries that source's own metric, tags and fields across the boundary and
makes the child placeable; a segment over two tables cannot sit inside
either region, so it runs above them. The single-source case keeps its
existing direct-scan path unchanged, which is what lets a one-table
segment still be pushed into its region.

Field names come from `tree_source_fields` on the rewritten tree, whose
source indexes are already dense in the order the children are built. That
helper also validates a repeated source against its own schema and refuses
a non-dense index set, which a second traversal would have to re-derive.
It becomes `pub` and is re-exported for the query crate to call.

Take the segment's time column from its own first source rather than from
the statement's first: a cut whose sources are `[1, 2]` would otherwise
name the time column of a table it does not read.

Cover the construction with a planner test and with a real-Mito
integration test that asserts both the exact values and the route: one
segment over the two tables, each child reading its own table through its
own raw series source. Without the route assertion a silent fallback would
pass on equal values.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): run a matching tree at the configured width

A matching tree's width came from its inputs: `matching_partitions` took
the selected source's count, or the maximum of every input's. In
standalone a cross-table statement arrives as one `MergeScanExec` per
table with one partition each, so `m / n` matched on a single worker while
the scans under it ran in parallel. The engine's
`execution.target_partitions` was reachable - `plan_fused_series` already
uses it for `series_scan` - but never reached this decision.

Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which
resolves a matching tree's width as `target_partitions.max(1)` for both
the same-source and the multi-input case. `to_execution_plan` keeps its
signature and its input-derived width, so a caller that has no configured
parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do
not consult the value at all: a per-series tree runs on whatever
partitions it was scanned with, and asking for a router there would only
add a shuffle.

Forward the value through `FusedSeriesExec` because `with_new_children`
rebuilds from the node's own children, which are the ones below the
router; rebuilding without it would resolve the width again from the
pre-router count and silently fall back to one.

Verified on a real standalone: the two-table segment now reports four
partitions with a `SeriesRepartitionExec` under each one-partition region
scan, where it reported one before, and the rows are identical.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): declare a keyed test frame's ordering, and order its keys

`plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`,
which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it
returns packs what `RawScript::ordering` reports, which is now the key-then-time
order this helper models. The node reads the ordering off the *plan* schema, so the
two disagreeing meant the router scheduled these fixtures as `Contiguous` while
unpacking validated them under the stricter ordered contract.

Build the schema through `frame_schema_with` with the ordering the stream actually
produces, so the plan and the runtime frames state the same contract.

That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with
both sides claiming `contiguous`, the ordered validation never ran. Running it
showed the matching fixture feeding key `b` before `a` within one partition, which
its declared ordering forbids. Reorder the partition; the two pieces of `host="a"`
still arrive in different partitions, which is what the test is about.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): declare the key order a two-phase raw scan actually keeps

`SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a
`TwoPhase` scan that is not reading columnar artifacts. The data that path
produces is ordered by primary key:

- candidates are merged and sorted ascending by the encoded primary-key
  column, then decoded to ids by walking that ordered stream without
  reordering;
- partition assignment is a deterministic TSID-to-partition map that
  appends in arrival order, so one series never spans two partitions;
- each assignment is drained completely before the next is taken, rather
  than emitted in completion order;
- the raw adapter's predicate buffering preserves input order and cannot
  let a later key overtake a held one, and two-phase records always report
  `ends_package: false`, so it never engages the package-tail machinery
  that legacy relies on.

Declaring the weaker order costs two things that are available: the frame
codec's `SequenceValidator` never ran its increasing-key check, and a
repartition owner could not use a key watermark to release an identity
before another input ended. It also contradicts the scanner's own
documented contract, which states the output is always ordered by
(primary key, time index) within a partition.

Report the order the path actually keeps, and reword the two ordering
errors that said "legacy" now that both modes can reach them.

Cover it with a test that forces two-phase mode and the non-prepared
fallback, asserts which of the two it ran, and then checks per-partition
key monotonicity, per-series timestamp monotonicity, single-partition
ownership, and the ordering the scan reports. The fixture writes three
rounds, uses two output partitions and a residual predicate, and keeps
one series above a batch so the multi-batch path is exercised.

The three existing two-phase raw tests asserted the old declaration. They
check that a series stays in one run and never recurs non-adjacently,
which the stronger order still guarantees; only their expected declaration
and their names said "contiguous", so those change and the substantive
assertions stay.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw match only by a key the computation keeps

`same_source` answers "this subtree reads one source", and a `MatchingLocal`
classification was taking that answer as licence to insert the router below
the fused computations, hashing the labels as they are at the raw stage.
Those are different facts. A math map drops the metric name and the
`__type__`/`__unit__` metadata, `timestamp` does the same, and every range
function drops them except `last_over_time` and `first_over_time` — so
`on(__name__)` over `abs(...)` routes by a key the very next node deletes.

Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side
routes `x` and `y` to different owners; both become the empty key after
`abs`, nothing re-partitions between the map and the match, and the pair is
silently lost.

Decide preservation separately from source identity. A node keeps the key
when it cannot remove one of the labels the key is built from; a `Window`
consults `RangeFunction::drops_metric_name` rather than assuming, so the
two functions that keep the name still route early. Anything not proven to
keep the key — a label rewrite, a histogram map, an unmodelled node —
does not establish preservation, and the tree is classified `SinglePartition`.

That is not a fallback that trades parallelism for correctness. When the
computation removes the labels the key is built from, every series on one
side reduces to the same empty key, so every left series matches every
right one: the match is whole-vector by construction and has no partitioned
form that preserves its result. One partition is the only placement that
answers it. Where the key does survive, the match stays partitioned.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): describe an admitted operand by what it produces

`try_plan_native_operand` handed the binary matcher the context saved
during scanning, which describes the operand's *input* layout. An admitted
call can change that layout: `histogram_quantile` folds a classic
histogram and drops `le`, so the scan context still names a tag column the
operand no longer produces, and default matching then builds a join key
over a column that is not there.

Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)`
declines whole-tree planning (two selectors over one table) and is then
admitted operand by operand; each operand drops `le`, and planning fails
building the join with `FieldNotFound { field: "le" }` against a schema of
`timestamp, field_0, host`.

Publish only the tag and field columns that the operand's own plan schema
actually contains, and drop `use_tsid`, which the produced output does not
carry. The check is against the schema the node returns rather than
against a list of functions that happen to shrink it, so a call added
later cannot reintroduce the gap by not being on a list.

Covered by a classic-histogram fixture with `host` and `le` and a float
bucket field, asserting each operand's context reports `["host"]` and that
the planned join key omits `le`. Reverting the change fails that test with
the `FieldNotFound` above.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the partitioned duplicate-group answer, with the gap named

`native_hybrid_binary_05_06_08` asserted that the right side's two
`job="other"` series are refused as a duplicate match group. On the
partitioned route they are not: the check is gated on `lhs.presence &&
rhs.presence` computed from the partition's own slices, so the partition
holding `other` sees no left-hand data and filters every point out.

Pin what the route actually answers, and state the three things that
constrain the fix, because each has already been tried and rejected here:

- the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a
  Prometheus-generated case where the left is absent at one point and the
  right carries three samples of one group there, and the answer is a
  value with no error;
- the scan cannot run unconditionally, for the same reason;
- the match cannot be collapsed to one partition to make both operands
  whole, because that gives up the parallelism this route exists for.

What remains is to let the verdict see whole-operand presence while the
partitions keep answering — a deferred check rather than a different
placement. This test now makes any change to the current answer visible
instead of silent.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a nested match instead of gathering it

A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` —
passes the logical admission check, because `independent_scan_pair`
recurses through `scan_operand`. The physical locality classification does
not go that far: it recognizes a match only when both operands read one
source each, so `same_source` fails on an operand that is itself a match
and the whole tree is classified `SinglePartition`. The fused computation
then ran on one partition while the scans under it stayed parallel, which
the existing planner does not do: it builds ordinary equijoins that
DataFusion can execute as partitioned hash joins.

Admission and placement are different questions, so they now get different
predicates. The tree is segmented when it is cross-table, not multi-input,
classified `SinglePartition`, and its root is a vector match with a nested
vector match directly under it. The inner match becomes a segment and the
outer one repartitions the evaluated operands by its own key, which is
what the segmenter already does for any exportable evaluated prefix; a
nested match whose two levels use different keys is routed twice, once per
key, rather than pretending one routing serves both.

The predicate stays narrow on purpose. Three-selector trees over distinct
metric logical ids are `multi_input` and keep their existing construction,
including the parent binding that construction produces. A same-table
nested match does not enter this branch either, so its prior admission is
unchanged.

The plan test's collector walks each fused node's native tree in
expression-child order and resolves each input's `source_index` against
that node's inputs, so a segmented plan is checked for the same property
as before — every selector occurrence bound exactly once, in expression
order — rather than being checked more weakly. It panics on a shape it
does not model instead of returning an empty list.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): prove a raw match key survives the computation

`preserves_matching_key` decides whether a subtree keeps the labels a
vector match routes by. Two arms answered yes without looking at the
field that decides it.

A binary with a scalar operand recursed into the vector side alone, but
arithmetic and a `bool` comparison drop the metric name while a filtering
comparison keeps it — the distinction execution already makes. And an
`Instant` recursed unconditionally, while `timestamp(x)` compiles to
`InstantValue::Timestamp`, which drops the name too; the check written for
the separate `Timestamp` node never covered that shape.

Both now read the field that decides: the operator together with
`return_bool`, and `InstantValue`. The labels checked are the whole
computed set — name, type and unit — because `drop_metric_name` removes
all three and default matching does not ignore the last two by itself.

Without this, `(x * 2) + on(__name__) (y * 2)` was classified
matching-local, the router hashed the raw names before the maps dropped
them, and a valid pair could land in different partitions with nothing
re-partitioning afterwards. The regression asserts the value is 10 and
that the tree is `SinglePartition`, computing the destinations the router
would have used (`x0` to partition 0, `y0` to partition 1 of two). With
the old arm restored the value assertion fails with `left: [10.0],
right: []`.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): find a nested match through the wrappers around it

Segmentation was gated on the root itself being a vector match with a
nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never
entered: the wrapper or the scalar broadcast sat where the match was
expected. Meanwhile the segmenter already walks children and leaves a
wrapper in the parent while cutting what it can carry, so no per-wrapper
case is needed — the search just has to look through them.

Replacing the root check with `carries_prefix` was too wide: almost any
multi-source tree contains something a boundary could carry, so ordinary
shapes like `some_metric unless on(tag_0) other_metric` were cut for no
gain and their inputs stopped being raw sources.

So keep requiring a nested vector match, and search for it anywhere in the
tree rather than at the root. Wrappers and scalar broadcasts are
transparent because the walk visits every child; the nested match itself
is still required, which is what keeps the plain multi-source shapes on
the plan they had.

`cross_table`, `!multi_input` and the `SinglePartition` locality
requirement stay, as before: they are what keeps distinct metric logical
ids and the same-table shapes out of this branch.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): let a router publish an identity whose chain is complete

`SeriesRepartitionExec` published an identity only once every input had
contributed it or ended, because an identity may be split across inputs. A
per-series region scan assigns each identity to one partition, so on that
shape the rule holds every identity until the other partitions reach EOF: a
region's whole raw selection is retained before the first series leaves.

The node now carries an `IdentityCompletion`, defaulting to the existing
merge. `CompleteDisjoint` publishes an identity as soon as its chain closes
and keeps the published identities charged in the pool, so a second chain
for the same identity fails the execution instead of emitting a partial
series as final. The declaration is the producer's, not something a consumer
infers from a node's type: publishing early against a producer that does not
own its identities would be wrong, so the conservative mode stays the
default and every existing constructor keeps it.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): choose a series-frame root's router from declared provenance

`SeriesFrameOutput` normalized every multi-partition frame root through a
router that merges identities, which is wrong for a root whose identities are
already disjoint: it waits for every scan partition to end before the first
series leaves.

The rule now selects `CompleteDisjoint` only for a root whose producer
declared that property. A raw root qualifies when `RawSeriesSourceExec` was
built through `to_per_series_execution_plan` - which also checks the child is
a `SeriesScanExec`, so the type alone is not taken as proof - and a direct
`SeriesRepartitionExec` root qualifies because it merged the identities
itself. Everything else, including a fused frame root, keeps the merging
router: a native label transformation can map two source series onto one
output label set, so its output identities are not exclusive.

The rejection of a key-only raw frame is unchanged and its reason is
restated: coalescing cannot produce such a frame, and a router needs explicit
labels to check that one key has one binding.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): report the labels `absent` names, not the table's columns

`fused_output_tags` answered `None` for `absent`, so the caller fell back to
the existing planner's column names. Those are the table's tag columns, which
is not what `absent` reports: `absent(m{host="x"})` carries the labels its
matchers name, and the compiled tree already knows them in
`BoundVectorOp::Absent { labels }`.

The label set is now read from that field: the named labels are kept in the
input's column order, so `absent(native_shapes{host="missing"})` over a table
with `host` and `le` reports `host` rather than `ts, val, host, le`. A label
the matchers name that is not an input column still answers `None`, because a
frame has no column to carry it; `count_values` and `histogram` are unchanged.

Caught by the native-versus-legacy value comparison: the extra column made
`absent` differ from the existing plan while every value matched.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): describe a frame root's emission from what it does

`FusedSeriesExec` declared `EmissionType::Final` for a frame output, but
`poll_output` hands a frame's evaluated chunk over while its input is still
arriving, so the declaration overstated the node to the optimizer. Only a
partial aggregate's state is emitted after EOF; every other output keeps its
input's emission type.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): compare the shapes the native route now admits with the existing plan

Removing the shapes a name list used to refuse put single-selector `sort` and
`sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route.
Nothing compared their answers with the existing plan, so the widening was
unverified.

The new case runs each shape both ways over one metric table. `sort`,
`sort_desc` and both `absent` cases compare the whole printed result, which
covers column names and values together. `histogram_quantile` compares values
row by row and asserts the one known difference instead of hiding it: a
segmented statement names its value column from the compiled tree (`val`)
where the existing plan names it after the expression that produced it. That
case is given a window long enough to produce a row, so it cannot pass by
comparing two empty results.

The `absent` cases caught a real difference: the native route reported the
table's tag columns rather than the labels the matchers name, which is fixed
in the preceding commit.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): publish a fused root's identities early when the tree keeps them

The root rule publishes an identity as soon as its chain closes only for a
root whose producer declared that each identity lives in one partition, and
it knew two such producers: a per-series raw scan, and a router that had
already merged its identities. A fused root was always treated as needing a
merge, so a region whose segment is a plain instant selection over one
per-series scan still held every identity until the other scan partitions
reached EOF - `sum(topk(1, m))` is that shape, and its segment is already
multi-partition with no label rewrite.

`FusedSeriesExec` now answers whether its output identities are disjoint, and
the rule asks it. The answer is narrow on purpose: one input, and a tree of
nothing but inputs and sample instant selections, over a per-series scan or
another fused node that already answered yes. Anything that computes labels
is excluded, because a timestamp or a pointwise map can map two source series
onto one output identity, and publishing early would then hand over one
identity's pieces as if they were different ones. A vector-to-vector binary
reads across series and is excluded too.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): segment a computed matching key's operands instead of gathering them

A vector match whose operands compute away one of the matched labels was
classified `SinglePartition` and executed on one partition. That is correct
for the bare tree - the raw names cannot route a match whose operands drop
them - but it also gave up the parallelism the operands still have:
`(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so
the computed sides can be routed by what they actually carry.

The planner now cuts those operands into segments of their own and lets the
parent route their evaluated outputs by the unchanged match key. The key
itself is not rewritten: `group_key` reads the labels present at that point,
so `On(__name__, host)` degrades to the `host` a computed operand still has.

Which shapes take this route is decided by one predicate shared with
`input_locality`, so the admission rule cannot drift from the locality
judgement: a matching site whose two operands are each proven single-source
computations, at least one of which does not preserve the key. A failed
preservation proof on its own is not enough - it can also mean the proof gave
up, as with a parameter that is a compound expression or an operand that is
itself a vector match - and the shared doc says so.

The same table read twice is admitted for this shape only. The rule that
declines several selectors on one table exists to keep folds such as
`sum(m) / m` off this route, not because one table cannot be read twice; the
compiler already gives each occurrence its own source index, and each cut
gets its own input. Distinct partitions are what the two operands need, and
they get them.

The regression executes both the same-table and the cross-table shape over a
four-host fixture and asserts that at least two output partitions produce
rows, so the result is not silently produced by one of them.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): take the early-publication guarantee from the scan's binding

The root rule publishes an identity as soon as its chain closes only when the
producer can promise that each identity lives in one partition, and the fused
answer to that question asked whether its input was a `SeriesScanExec`. The
node's type is not that promise: a scan binds a set of logical tags and decodes
those into the series' labels, and `try_to_named_series_scan` requires the
caller to pass complete tag names but does not check it. A binding that names
only some primary-key columns maps two stored series onto one output identity,
and early publication would hand one identity's pieces over as different ones.

The answer now reads the binding: the label source has to be `PrimaryKey`, and
every primary-key column has to be among the bound tags. A source that carries
its labels explicitly is not this shape and answers no, as before.

Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is
false, full coverage is true, an explicit source is false - and by tests for the
predicate's two other parts: a tree that computes labels and a tree that is not
one instant selection over one input both answer no, and a gated execution
shows a completed identity leaving a fused node while the other input is still
held.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): drop the checks the types and decoders already guarantee

Two re-checks ran inside the repartition node over facts its inputs had
already established, and both cost real work on every query.

A raw merge scanned each chunk's timestamps for strict increase, but
`RawTimestamps` cannot hold an unordered buffer: its field is private, its only
constructor validates, and the accessors are read-only or order-preserving, so
no safe caller can build the value this check defends against. The check is
gone; the identity, label and shape checks around it stay, because those
compare different producers' pieces rather than one chunk's own timestamps.

An owner rebuilt each chunk's identity to prove a chain does not change it, but
the decoder that hands the chain over already refuses that: `SequenceValidator`
holds one open key and rejects a key or metric change before the chain closes,
`EvaluatedSequenceValidator` does the same for labels, and the input task that
assembles chains only reads from those decoders. What stays is the identity the
owner needs anyway to pick a destination, now built by the named functions
rather than inline.

Where a schema was needed, the node built an empty stream of its own, wrapped it
in a packer and read the packer's schema back. The frame module already answers
that question from the declaration, so the fake stream is deleted along with its
half of the roundtrip.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recursively segment native matching plans

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): derive native schemas without legacy probes

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recognize complete metric scan identities

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23)

* fix(promql): release completed disjoint identity history

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c)

* test(promql): assert native aggregate schemas independently

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): identify internal metric keys by reserved id

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep parallel sources and derive group match labels

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe the frontend fused root in scan-parallelism checks

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): release duplicate evidence immediately on abort

Signed-off-by: discord9 <discord9@163.com>

* test(promql): preserve operand identity across matching repartition

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): charge the final duplicate verdict until it is reported

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe native scans across frontend and region plans

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): drop dead test helper and clippy lints

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): select limitk by canonical label order

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep data-free subtrees in the parent segment

Signed-off-by: discord9 <discord9@163.com>

* test(promql): prove the duplicate error crosses matching partitions

Signed-off-by: discord9 <discord9@163.com>

* test(promql): drop debug prints from the segmentation test

Signed-off-by: discord9 <discord9@163.com>

* revert(promql): keep limitk's Prometheus input-order semantics

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): satisfy clippy in fused root schema

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-04 14:12:56 +08:00
discord9 6eff9f5804 test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 78be1b1138 test(promql): regenerate independent matcher and range results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 20:27:32 +08:00
discord9 8dbf08d252 feat(promql): execute independent selector scans in fused binary trees
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 20:10:32 +08:00
discord9 c3dd11bf48 test(promql): record shared-selector SQL regression results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 14:31:27 +08:00
discord9 2c3dace6b2 test(promql): align frame ordering and SQL result sorting
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 13:47:42 +08:00
discord9 4748081579 feat(promql): fan out a shared raw scan to bound binary inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 13:47:42 +08:00
discord9 55e59f6c91 fix: update semantic_graph expected error code for typed error preservation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 20:26:18 +08:00
dennis zhuang 5c4d9eb4f8 fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE

The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.

Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop unused lazy_static from index and avoid typo-flagged test tokens

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: name probe extraction with verbs and document CharClass

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: rebuild legacy fulltext bloom indexes and probe lower(col)

The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.

Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover legacy bloom index rewrite by compaction in the compat case

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: log outdated fulltext bloom indexes in the consistency check

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: explain the unrecorded fulltext bloom version

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-14 08:07:24 +00:00
Yingwen 00aa65de7a fix(query): guard deep expressions during query planning (#9564)
* fix(query): avoid speculative clones in constant normalization

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(query): reject deeply nested expressions before optimization

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-14 06:49:35 +00:00
dennis zhuang 2a70194930 feat: add per-column COMPRESSION option for SST columns (#9426)
* feat: add per-column COMPRESSION option for SST columns

CREATE TABLE accepts COMPRESSION WITH (type = 'zstd', level = N,
page_size = '<size>') on a column. The options are stored in the column
metadata, carried through gRPC column options, shown by SHOW CREATE
TABLE, and applied by the mito SST writer and bulk part encoder as
parquet column properties. Unset fields keep the SST defaults. Nested
types are rejected.

A page size below the dictionary page limit also lowers the column's
dictionary page limit: parquet sizes write batches by the dictionary
budget and keeps that size after falling back to plain encoding, which
would otherwise ignore the smaller data page limit.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: validate column compression options before persistence

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: reject compression overrides on metric logical tables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-13 08:03:49 +00:00
Weny Xu 287ced2de2 fix(mito2): ignore missing inverted indexes in old SSTs (#9545)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 03:18:37 +00:00
shuiyisong 2df33380a7 feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: rename cluster to global

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: update sqlness result

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(catalog): require information-schema table scopes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-12 08:49:33 +00:00
dennis zhuang f3a0841f4f fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations

- Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL
  comparisons with NaN are false except !=.
- min/max: skip NaN unless the whole group is NaN. DataFusion's grouped
  min/max let NaN replace and be replaced by neighbours, so the result
  depended on input order.
- topk/bottomk rank NaN last.
- round() rounds ties up and returns NaN for a zero step, like Prometheus.
- sqrt() of a negative number is NaN instead of an error.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round() without a step rounds to integers

The planner passed a zero step for single-argument round(), which now
yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding

sqrt resolves through the session's builtin functions, not the UDF branch.
Distributed plans are decoded with the serializer's session, which did not
know prom_min/prom_max.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover NaN and rounding semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the nested count rewrite for PromQL min/max

count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin
min/max names. prom_min/prom_max return NULL for the same groups, so the
rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt,
topk and min/max changes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): register prom_min/prom_max in the function registry

Distributed plans split an aggregation into state and merge steps only for
functions whose state function exists in FUNCTION_REGISTRY; prom_min and
prom_max lived in the promql crate, so min/max stopped being pushed down to
datanodes. Move them to common-function and register them before the state
functions are derived.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update sqlness results for PromQL float semantics

Plans and column names now show prom_min/prom_max, the NaN guards of
comparisons and sqrt, and the NaN sort key of topk. round() results follow
Prometheus' formula. Distributed min/max keep the state/merge split.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep column names and NULLs in IEEE comparisons and sqrt

Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed
TQL output column names and turned NULL operands into true for `!=` and
false for the bool forms. Implement comparisons and sqrt as PromQL UDFs
that propagate NULL and report the column name of the expression they
replace; prom_min/prom_max report min/max column names the same way.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: restore column names in sqlness results for PromQL float semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): share the prom_min/prom_max groups accumulator with tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep distinct names for prom_min/prom_max

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:10:18 +00:00
dennis zhuang 845fc0af9c fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step

`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover the default subquery step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover the default subquery step in a range query

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:10:07 +00:00
discord9 28ec3ff5aa fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 12:28:47 +00:00
discord9 32e8467cda fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): use current DataFusion plan downcasts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): match null rendering in join regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh union plan for unknown scan partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh shared plans after distribution rule removal

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover semi joins with remote dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen partitioned merge scan regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove unused merge scan test helpers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): measure merge scan hash repartition cost

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt merge scan deserialization to two-arg constructor

Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): adapt nested merge scan integration constructor

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 11:18:46 +00:00
fys 9a57e1195d feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com>
2026-10-10 08:31:18 +00:00
dennis zhuang b913be2af7 fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries

An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): keep encode_substrait on a valid range query

A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): plan wrapped range-vector roots and reject a zero subquery step

TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover wrapped range-vector roots and a zero subquery step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): default subquery step and reject range vectors in range queries

A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover root subquery default step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): align root subqueries with subquery_child_window

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): return millisecond timestamps for range-vector results

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): keep the time index timezone and name its table in errors

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 07:04:02 +00:00
discord9 178a0021b8 fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds

Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.

The metasrv-side validate_flow_options remains as a defensive check.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): use DISTINCT output columns as sink key for streaming flows

When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.

Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): preserve extension options and complete DISTINCT sink identity

Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): align meta unknown option status assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): use valid aliases in DISTINCT sink regression SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): record complete DISTINCT sink SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:45:11 +00:00
dennis zhuang a8e293fd79 fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch

Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update flow_tql error for a negative TQL end time

A negative end now parses and fails the end >= start check instead of
the timestamp conversion.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): reject evaluation times outside the millisecond range

The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop an unused import

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round query times to milliseconds like Prometheus

Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round the float fraction before converting to nanoseconds

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: skip the overflow case where SystemTime cannot hold it

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 05:26:21 +00:00
Niladri Adhikary 0dde3a7b70 fix(promql): return no point for zero-interval irate/idelta (#9455)
fix(promql): irate/idelta return no point for a zero-ms sample gap

Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
2026-10-09 10:03:36 +00:00
fengyue-xve 764ab8bc1a test(promql): cover label_replace normalization with distinct series (#9485)
The duplicate-label-set rejection must not fire when a label rewrite keeps
every series distinct through another label. Normalizing mixed histogram
boundary representations (`10.0` -> `10`) while `instance` still differs is
the false positive reported in #9444. The check already accounts for it
since it became a per-timestamp row count instead of a planning-time
rejection, but no fixture covered the positive case. Add one, together
with the aggregation variant that produces a genuine duplicate and must
still fail.

Signed-off-by: fengyue-xve <1759519137@qq.com>
2026-10-09 09:59:20 +00:00
dennis zhuang 6da5352c53 fix(promql): keep vector labels when a computed scalar is on the right (#9478)
* fix(promql): keep vector labels when a computed scalar is on the right

`metric + time()` or `metric * scalar(other)` planned the join with the
scalar side's context, which has no tags, so every series collapsed into
one unlabeled series. Use the vector side's context when the right operand
is scalar-typed.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): update results for labels kept with a computed scalar on the right

v * pi() and v + time() now keep the vector's labels and time index name,
as in Prometheus.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): qualify columns of an unnamed operand in vector joins

A comparison filter's projection has no table name, so after the join its
time index resolved unqualified and was ambiguous when the other operand had
the same column. Also keep the empty-metric branch unchanged for scalar
left operands.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover comparison filters joined with a scalar

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-09 07:16:42 +00:00
discord9 0d2cb1aa7e fix(promql): align subqueries to the absolute evaluation grid (#9400)
* fix(promql): align subqueries to the absolute evaluation grid

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(promql): streamline subquery comments

Remove redundant arithmetic narration and restated test comments. Keep grid, fallback, validation and overflow contracts, and preserve all regression tests and assertions. Regenerate SQLness output only for comment echoes.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-09 03:50:54 +00:00
fys fe8a647bc8 fix(json2): support settings in ALTER TABLE ADD COLUMN (#9472)
* fix(json2): support settings in ALTER TABLE ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(sql): reuse Column for ADD COLUMN definitions

Signed-off-by: fys <fengys1996@gmail.com>

* chore: better method name

* refactor(sql): share column parsing between CREATE TABLE and ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* chore: simplify tests

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): accept trailing commas in ALTER TABLE ADD COLUMN

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): preserve ALTER TABLE ADD COLUMN compatibility

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-09 03:17:22 +00:00
fys c46fbde9ac fix(json2): preserve metadata when adding columns via alter table (#8696)
* fix(json2): support adding columns via alter table

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): scope missing-column null handling to JSON2

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-08 09:23:25 +00:00
dennis zhuang 526dd48bda fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: verify compound filter semantics and last-row selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: consolidate label filter regression coverage

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: separate filter evaluation and timestamp adaptation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: inline predicate and timestamp tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: keep direct IN predicates outside simple filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: coerce partition bounds before simple filtering

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover partition bound overflow after timestamp widening

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 07:43:25 +00:00
dennis zhuang b78dc4b33b test(sqlness): wait for mirrored inserts before flushing streaming flows (#9424)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 02:30:37 +00:00
Yingwen 97913b4b3a fix: coalesce equal histogram quantile bucket boundaries (#9451)
* fix: coalesce equal histogram quantile bucket boundaries

Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.

Fixes #9443

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: keep duplicate infinity buckets in one histogram group

Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.

Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: avoid buffering histogram groups with invalid trailing bounds

Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-07 07:18:25 +00:00
Yingwen 3a4c24a863 fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: explicitly enable v2 series scan in sqlness configs

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: regenerate sqlness results for legacy series scan default

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-02 12:09:16 +00:00
dennis zhuang 78a7b93292 fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics

- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): align label_replace and label_join edge cases with Prometheus

- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): join absent labels as empty strings in label_join

concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): skip rows without a sample in timestamp()

timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 01:42:11 +00:00
dennis zhuang 218000e21b fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns

col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): resolve remote read matcher labels as unqualified columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover same-name columns differing in case and without()

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-29 08:13:28 +00:00
DeviousCardiandClaude Opus 5.5 d6474b961c fix(promql): apply offset to subquery evaluation window (#9364)
* fix(promql): apply offset to subquery evaluation window

`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.

Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.

`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.

Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.

Closes #9330

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* docs(promql): correct the subquery-offset rationale and pin the histogram shift

Follow-up to e132c267, which stated two things inaccurately and left one
path untested. No behaviour change beyond a conditional clone.

The `SeriesNormalize` carrier was described as existing "so the offset
survives distributed planning", implying standalone is unaffected. That is
wrong. `local_offset` is read on the substrait decode path
(`range_manipulate.rs`, `instant_manipulate.rs`), and GreptimeDB routes
PromQL plans through `MergeScan`/substrait in standalone too -- the
standalone results `promql/encode_substrait.result` and `precisions.result`
both show `MergeScan [is_placeholder=false, remote_input=[...]]`. Deleting
the carrier therefore empties subquery-with-offset results in standalone as
well, while `count_over_time_subquery_with_offset` keeps passing because it
asserts the pre-serialization plan. Say so at the node, so the next reader
does not remove it believing standalone is safe.

The carrier is also not semantically inert: `SeriesNormalizeStream::normalize`
shifts native histogram `start_timestamp` payloads by `offset`. On the
subquery path that runs on a computed inner result, on top of any offset the
inner selector already applied, and nothing pinned it. It is consistent --
`RangeManipulate` moves the millisecond time index forward by the same
`offset_ms` -- so the distance from a histogram's start timestamp to the
sample carrying it, which reset/rate detection reads, is invariant. Add
`subquery_offset_shifts_histogram_start_and_time_index_together`, which
executes the subquery's node stack over native histogram samples for a
zero and a non-zero inner offset. It cannot be a sqlness case: native
histograms are a struct column with no SQL type or literal
(`sql_data_type_to_concrete_data_type` rejects structs) and the sqlness
runner speaks only MySQL/Postgres, so such rows only arrive over
gRPC/remote-write v2.

The divergence note blamed sub-step offsets. Measured, that condition is too
narrow: an unaligned evaluation timestamp alone diverges, with no offset at
all -- `sum_over_time(fine[20s:10s])` at t=57 samples 47s and 57s here
against Prometheus's 40s and 50s. The real condition is that
`(start - offset)` is not a multiple of the subquery step, and it is
pre-existing: the committed `tql eval (359, 359, '1s')
sum_over_time(metric_total[60s:10s])` case is already an instance of it
(`359 - 60 + 10 = 309`). Reword the comment and the sqlness header
accordingly. A sub-step offset stays accepted: `foo[20s:10s] offset 5s` is
valid PromQL, erroring on it would be a regression and would not close the
gap anyway. This repo has no PromQL-compatibility documentation page, so
there is nowhere else to record it.

Also stop cloning the series key columns when there is no offset, and stop
the `offset 0s` sqlness comment implying its parser error is evidence about
this fix -- the same message comes from a `u32` overflow in
`promql-parser`'s shared duration check (`offset 9999999999d`).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* refactor(promql): trim subquery-offset comments and tests per review

Shorten the planner and sqlness comments to the semantic contract, build
`SeriesDivide` with a clone instead of an `Option`, and drop the
normalize.rs histogram test and the `offset 0s` parser case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* fix(promql): record the subquery offset as the range fold offset

The subquery's `RangeManipulate` now shifts the payload timestamps by the
offset, so `predict_linear` over an offset subquery must recover the
evaluation time with that offset instead of 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>

---------

Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 03:38:43 +00:00
discord9 28e415a103 feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors

The planner previously dropped the `at` field of selectors (`at: _`),
so `some_metric @ 300` silently returned step-following values instead
of the samples anchored at the fixed timestamp.

Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch`
semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply
`offset` to the anchor, rewrite the selector offset to
`eval_start - anchor`, scan only the anchored window, then report the
same window at every evaluation step via a grid-wide-replay
`InstantManipulate`. `@ start()` / `@ end()` resolve against the
statement's evaluation range (`stmt_start` / `stmt_end`), which subquery
planning does not rewrite.

Timestamps before the Unix epoch are accepted; unrepresentable ones are
rejected with `AtModifierTimestampOutOfRange` instead of wrapping.
Selectors without `@` are planned exactly as before.

Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): center predict_linear on the evaluation timestamp

predict_linear_impl used the window's last sample time as the evaluation
timestamp, and the UDF took only (ts_range, value_range, t) with no channel
for the current instant. After a range selector is folded for @ (or with an
offset) the same window is replayed at every step, so the prediction stayed
constant at the anchor's answer; even a plain window ended before the step
produced a stale value.

Give prom_predict_linear a 4th argument carrying the step's evaluation
instant (ms timestamp), derived in the planner from the row's time index
plus the offset the window was folded with (at_offset for @, offset_ms
otherwise, recorded on PromPlannerContext). The regression is centered on
that instant, matching Prometheus' use of enh.Ts. The mixed
float/native-histogram path forwards the same argument.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): tighten @ modifier planner and predict_linear

- range_fold_offset is always a concrete millisecond offset, not an
  Option; the single-step replay helper no longer wraps an infallible
  plan in Result.
- predict_linear's eval timestamp is always cast to Timestamp(ms) at the
  call site, so the UDF drops its dead Int64 branch and the extra
  func_name argument.
- Drop two redundant comparison cases from the at_modifier sqlness test
  and fix the lookback window comment to the half-open (0s, 300s].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): accept parse-time rejection of @ on Windows

`@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime
holds it and the planner rejects the anchor it cannot represent, but a
Windows SystemTime tops out near 1.8e12 seconds, so the parser's
checked_add fails first and the same literal is rejected while parsing.
Accept either rejection path so the test passes on both platforms.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): avoid replaying label_join over rewritten series keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): narrow anchored range call promotion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): address @ modifier review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): satisfy super import format check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): address at modifier review follow-ups

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 09:38:12 +00:00
discord9 4ffd93d8dc fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning

Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch)
below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The
DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then
dropped or rewrote that fetched coalesce, losing the global limit. Root-caused
and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo
pins to that fix commit and adds regression coverage.

- pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix)
- global_limit.rs: adapt to new DF API (input_distribution_requirements,
  child_distribution, replace_children+Recompute); accept HashPartitioned and
  KeyPartitioned as partitioning to restore; add regression unit tests
- sqlness: extend standalone common ssts and refresh distributed ssts_limit
  result for the preserved fetch plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: drop dead HashPartitioned compat arms and test wrapper

- remove HashPartitioned compatibility matches (pinned DF only emits
  KeyPartitioned); drop their #[expect(deprecated)] attributes
- remove single-use agg_with_limit helper whose seed limit is always
  overwritten by the soft-limit transform
- restore Cargo.lock dependency edges unrelated to the DataFusion pin
  bump (cargo update --precise had downgraded 26 unrelated packages)

Addresses oracle ablation review.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(deps): pin DataFusion to merged fetch preservation fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 03:35:36 +00:00
dennis zhuang 357c935804 fix: make vector aggregates work with GROUP BY and partial aggregation (#9338)
* fix: make vector aggregates work with GROUP BY and partial aggregation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: make partitioned vec_avg case distinguish weighted averages

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 08:45:40 +00:00
discord9 abedeb2bea fix(query): keep count_values generated label in enclosing expressions (#9223)
* fix(query): keep count_values generated label in enclosing expressions

`count_values("v", m)` projects the generated label as a real output
column, but did not register it in `ctx.tag_columns`. Enclosing
expressions (abs/round/+1/topk/label_replace/vector join) rebuild their
projection from `ctx.tag_columns` and silently drop the label.

Register the generated label in `ctx.tag_columns` after the projection,
and give it the same qualifier as other tag columns so qualified
references resolve. PromQL overwrites an input label with the same name,
so drop any existing tag with that name first to avoid duplicate column
ambiguity.

Fixes https://github.com/GreptimeTeam/greptimedb/issues/9181
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-1

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): use total_cmp in count_values test helper

Silence clippy::needless_borrow on partial_cmp(&right.1); f64 sorting
uses total_cmp, matching the other planner test helpers.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): keep count_values generated label as sink table primary key

FindGroupByFinalName::f_up only renamed a group key when the projection
aliased the key column directly. count_values now projects its generated
label as a unary UDF over the sampled column
(prom_float_to_string(value) AS label) while the aggregate still groups
by the raw column, so the name match failed and the sink table lost the
label from its PRIMARY KEY, demoting it to a DOUBLE value column.

Allow a projection above the aggregate to rename a group key by deriving
its output from a single group-key column (unary scalar function or cast
over that column only). Multi-column expressions, case, aggregates,
windows, subqueries and literals are still rejected so a computed column
cannot be mistaken for the group key.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): group count_values by the formatted sample value

The numeric count_values branch grouped by the raw sample column and only
formatted the value into the generated label in the post-aggregate
projection. Two distinct raw values that collapse to one label text
(e.g. BIGINT 9007199254740992 and 9007199254740993, both 9007199254740992
in Float64) were split into two groups, each emitting the same label set
at one timestamp, violating Prometheus' unique-label-set-per-timestamp
invariant.

Return the formatted value expression (prom_float_to_string, with a
CAST to Float64 for non-Float64 inputs) as previous_field_expressions so
the aggregate groups by the same expression that produces the label,
mirroring the existing mixed float/native-histogram precedent.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): drop needless borrow in count_values test helper

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): don't replace group key with derived unary expr

FindGroupByFinalName treated any unary expression of a single group
column (scalar fn, Cast, TryCast) as a rename of that group key and
swapped it in as the sink primary key. A derived expression such as
lower(host) is many-to-one, so distinct groups (HOST_A vs host_a) would
collapse to the same primary key and be silently deduplicated.

Narrow the matching to direct name-matched aliases of the actual group
expression only, and drop the derived-unary path (is_unary_expr_of_column)
and the allow_derived flag. count_values already groups by the formatted
expression name, so it is unaffected.

Add a regression asserting host survives and host_lc does not replace it
under both optimizer settings.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-24 08:18:41 +00:00
dennis zhuang d59725b04a test(sqlness): run environments concurrently and drop redundant restarts and sleeps (#9333)
* test(sqlness): run environments concurrently with external backends

The runner forced both environment and instance parallelism to 1 whenever
etcd/PG/MySQL or an external Kafka was set up, so the standalone and
distributed environments ran one after the other.

Only distributed uses the kv backend, and the two environments use
different Kafka topic prefixes, so they can run at the same time. Keep one
instance per environment, since instances would share the metadata table
and Kafka topics. Runs with a runner-managed Kafka, an external server, a
test filter, or `-j 1` stay fully serial.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(sqlness): drop redundant restarts and sleeps

- flow_rebuild: drop the R2 block, which repeats R1 from the same state,
  and the restart in R5, whose result R1 already asserts.
- flow_advance_ttl, flow_view, ttl_instant: drop sleeps that no assertion
  depends on.
- flow_basic, flow_null, flow_call_df_func: drop the bytes_log section
  (covered by flow_insert), a duplicate state_size query, an unchecked
  insert, and flushes that run with no new data.
- alter_table_options, skip_wal: share restarts between independent tables.
- session_skip_wal, copy_skip_wal: check every case after a single restart.
  Tables copied or inserted with skip_wal = true first get a WAL row that is
  truncated, so the check that truncated WAL entries are not replayed stays.
- region_statistics: wait once for the statistics of all three tables.
- build_index_table: fold its index_size checks into
  build_index_table_restart, which runs the same fixture and waits.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(sqlness): restore session_skip_wal and copy_skip_wal

Checking every case after a single restart dropped two state transitions
the original cases cover: recovering a region whose memtable only held
skipped rows, and writing to the recovered region before restarting again.
Restore the original cases.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(sqlness): restore the one-restart check in flow_auto_sink_table

#5112 checked the auto-created sink before a restart and the flow and sink
after it. #5987 moved the restart in front of the first SHOW and added a
second one. Flow recovery only creates the sink when it is missing, so the
second restart recovers from the same persisted state as the first.
Restore the original before/after check with one restart.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(sqlness): keep one instance per environment with external store addresses

Instances of the distributed environment share the etcd behind --store-addrs, so run one instance per environment when it is set, same as --setup-etcd.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(sqlness): restore the false-branch batch in flow_basic and assert it

The kept batch (20, 22) is all above the threshold. The second batch (10, 23) is the only input that hits the false branch of the CASE and makes the flow compute again. Restore it and check the result, which the original case never did.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 07:03:50 +00:00
dennis zhuang affc0a1b1d fix: align ordered aggregate state type with the accumulator output (#9340)
* fix: align ordered aggregate state type with the accumulator output

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: reject mismatched aggregate states and keep hard-ordered aggregates unsplit

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: keep WITHIN GROUP aggregates splittable and relabel state fields by position

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 07:02:14 +00:00
dennis zhuang def0c2ec5c fix: only push down aggregates grouped by the partition columns themselves (#9337)
* fix: only push down aggregates grouped by the partition columns themselves

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: keep grouping sets on the frontend for partitioned tables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 06:49:52 +00:00
dennis zhuang b9f991502c refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 05:38:54 +00:00
dennis zhuang cc1be82858 fix: merge every state row in geo_path and json_encode_path (#9339)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 02:12:11 +00:00
dennis zhuang 88197f4019 fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings

A vector-vector binary operation projected one operand's whole tag set and
inner-joined without any cardinality check, so `on()`/`ignoring()` did not
reduce the result labels, `group_left`/`group_right` changed nothing, and a
non-unique match group produced a cross product that PromQL cannot represent.

Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the
matching labels, `ignoring(...)` drops them, and a group modifier keeps the
many side's labels plus the `group_x(...)` labels taken from the one side.
A label the one side does not carry is deleted from the result. The reduced
label set no longer identifies the operand series, so `__tsid` is dropped
from the context on this path.

Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)`
window and a scalar UDF that fails the query on a repeated group: on the one
side before the join, and on the result labels after it, matching where
Prometheus raises each of its three errors. Series are unique by their whole
tag set, so the window is only planted when the match keys drop a tag; plain
arithmetic and `on(<all tags>)` plan exactly as before.

Closes #9207, closes #9208, closes #9209.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): keep __tsid when the result labels are an operand's whole tag set

Deriving the result labels dropped `__tsid` from the context unconditionally,
so an enclosing operation fell back to joining on the tag columns even where
the column still identified the result series.

Keep it when every result label comes from one operand and covers that
operand's whole tag set: no other operand value reaches the labels, and the
matching gives each of its rows a single partner, so its `__tsid` is still one
per result series. That is the common `on(<all tags>)` and bare `group_left`
shape; a matching that actually drops a tag still clears it.

The column is re-qualified as the result's own, which is how the enclosing
expression and the context look it up.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the match group count column unambiguous

The cardinality check aliased its row count to a fixed `__promql_match_group_count`.
An operand carrying a label of that name made the window output two fields with
the same name, and planning failed with "Schema contains qualified field name
collide_right.__promql_match_group_count and unqualified field name
__promql_match_group_count which would be ambiguous".

Pick a name the operand does not already have, the way the `or` operator
allocates its match key columns.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover a match group spread over several regions

The cardinality check runs above the merge of the region scans, so it counts a
match group globally. Nothing pinned that: every table in these cases holds a
single region, and a check evaluated per region would pass them all.

Partition the operand on a column outside the match keys, which puts the two
series of one match group in different regions, and assert both the pre-join
and the post-join check still reject it. The case runs in the distributed
environment too, where the regions sit on different datanodes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): group an outer aggregate on the labels the operand kept

`by`/`without` planning topped up a missing grouping column by walking down to
the table scan and re-projecting it. That is right for a column the scan pruned
for efficiency, but the labels a matching modifier deletes are also absent from
the operand's output, and they were restored the same way:

  sum without(host) (a / on(host) b)

`on(host)` leaves the operand with `host` alone, so the sum covers everything
and Prometheus answers `{} 10`. Instead `device` came back from the scan under
`a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for
`sum by(device)` of that operand, which has no `device` to group on at all.

Take the grouping labels from the operand's own label set rather than from the
row keys of the scan beneath it. A label pruned from the plan is still in that
set and still gets restored; a label the operand dropped is not.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): drop the unreachable aggregation tag top-up

`by`/`without` planning could restore a grouping column that the plan no longer
carried by rewriting the scan underneath it. Once the grouping labels come from
the operand's own label set, there is nothing left for it to restore: a scan
projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds
matcher columns to that set), so a label in the set is always in the schema.

Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the
standalone and the distributed environment.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): drop cases that guarded the removed tag top-up

Three plain selector aggregates were there to show that restoring a pruned
grouping column still worked. With the restore gone they only repeat what the
aggregate cases already cover. Also fix a comment that still said the metric
engine scan prunes tag columns: it projects every label of the operand.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): note the duplicate a propagated matcher hides

A matcher copied onto the one-side operand removes groups without a partner
before the cardinality check sees them, so a duplicate in such a group is not
reported. Prometheus checks every group of the one side and fails the query.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 11:08:18 +00:00
fys 5a78e07be6 fix(json2): restrict JSON2 type hints (#9316)
* fix(sql): restrict JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): allow equivalent 64-bit aliases in JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* fix(sql): support UInt64 conversion and preserve JSON2 hint aliases

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(sql): remove redundant JSON2 type hint normalization

Signed-off-by: fys <fengys1996@gmail.com>

* fix(datatypes): restrict JSON2 hint types in JsonSettings::try_new

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-23 09:19:18 +00:00
Lei, HUANG d0f8f4b80c feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 14:12:24 +00:00
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-22 12:56:26 +00:00
dennis zhuang 25bca4609f perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 11:25:50 +00:00
Lei, HUANG b5199bc59a feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: gate Jev filtering behind an opt-in Cargo feature

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: verify Jev registration with default features

Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: clarify Jev concurrency scope and stabilization work

Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: enable renamed ai-functions feature by default

Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: remove extra AI feature-off checks

Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: rename AI feature to ai_functions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 03:03:23 +00:00
LFC 16978cf6c2 feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2

Signed-off-by: luofucong <luofc@foxmail.com>

* fix: remove unused annotation context import

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-22 02:39:36 +00:00