906 Commits

Include renames

Author	SHA1	Message	Date
discord9	fe2e364d23	feat(promql): execute split native aggregates as partial state over a state router (#10147) ... * feat(datatypes): add the shared aggregate-state representation A frame that carries partial aggregate state has to say which aggregate it is state of and which accumulator components it carries, or a consumer can read a sum state as an average one. These types are that statement: the kind, the ordered component list, one position's values, and the schema that validates the two against each other. Nothing uses them yet; the frame content that carries them follows. Signed-off-by: discord9 <discord9@163.com> * feat(datatypes): carry aggregate state as a third frame content A state frame keeps the series-major columns a frame already has — the key, finality, grid spans and labels — and puts each aggregate field's accumulators in one list of component structs under `fields`, the shape a classic histogram payload already uses. An absent list item is that field having no contribution at that position, so a hole stays a hole instead of becoming a zero, and a group state — which carries no component at all — still says per field where it was folded. The reader keeps typed handles into the batch rather than expanding the state into rows and rebuilding it, so the merge above it reads the arrays where they lie. Signed-off-by: discord9 <discord9@163.com> * feat(common-query): pack and unpack aggregate-state series frames The packer takes what a partial aggregate produces — one group, its carried positions and its per-field state — and the unpacker hands back the row with typed handles into the batch. That asymmetry is deliberate: export columnarises once, import must not rebuild the state to read it. Validation mirrors the evaluated pair: the declaration is parsed, contiguity required, the empty batch decoded before the stream is returned, and kind, component order, grid and field count are each checked with the part that disagreed named. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run native partial and final aggregates An aggregate node used to compute only `Complete`, refusing the stages a split aggregate needs. It now folds evaluated chunks into kernel state for `Partial` and merges that state into values for `Final`, so the compiler's `Partial -> Final` plan has a runtime behind it. The executor gains a third chunk kind: an input leaf accepts raw, evaluated or state chunks according to what its binding produces, never guessing from a schema, and a state chunk cannot be handed to a node that reads values. The state binding now carries the aggregate kind and its ordered components, derived from the kernel's own state fields, so a sum state cannot be merged as an average one. The sibling router refuses aggregate state explicitly for now; the node that routes it follows in its own change. Signed-off-by: discord9 <discord9@163.com> * feat(promql): route aggregate state by its group labels A state row is one group, complete as one producer wrote it: there is nothing to chain and nothing to merge, so the existing repartition node — whose invariants are all about assembling a key's chain and waiting for every input to close — is the wrong place for it. This node does one thing: send each row to the output its canonical labels name, so every contribution of a group reaches one owner and the merge above sees all of them. The read belongs to the outputs that were claimed, not to the plan, so dropping every output stops it and returns what it reserved. Signed-off-by: discord9 <discord9@163.com> * feat(promql): execute partial aggregates as state frames A fused node now reads its inputs by what they produce — raw samples, evaluated values, or aggregate state — instead of inferring the kind from a schema, and a partial root answers state rather than rows: the executor's state chunks are packed into the aggregate-state frame the merge above consumes. The temporary partial-fold bridge goes with it: no marker field, no side channel fold, no compatibility accessor. A tree that folds on the side is exactly what this change exists to remove, and the aggregate merge that replaces it is a separate node. The state path derives its mode and its binding from one reading of the root, so the two cannot disagree, and no production path panics on a malformed tree. Signed-off-by: discord9 <discord9@163.com> * feat(promql): split native aggregates into partial and final trees The previous commit made a `Partial` aggregate export its state, but left `AggregateFinalize` in place: it still called `FusedSeries::try_new_partial` and `partial_fold`, which that commit deleted, so the crate did not compile. Replace the bridge with the two trees that run a split aggregate: - `native/split.rs` owns the cut. It moves out of `aggregate_finalize.rs` unchanged in structure, and `Split::into_trees` binds the two halves: the prefix under a `Partial` aggregate answering aggregate state, and a `Final` aggregate consuming that state with the suffix above it. - The suffix is narrower than the old merge admitted. Only point-wise maps and a binary operation with a constant side may sit above the cut; `topk`, the classic-histogram functions, a second aggregate and a vector matching are declined, because the groups a `Final` aggregate owns are split across partitions and each of those reads a whole vector. - `FusedSeries` gains `try_new_state_export` (a `Partial` node declares its own state frame schema rather than taking a caller's), `split_aggregate` (declining shapes become `None`, not an error), `reads_aggregate_state` and the state-input path of `to_execution_plan_with_parallelism_option`, which routes state through `StateRepartitionExec` and lets a cross-series tree read state from more than one partition. - The runtime executes the `Final` stage: the guard that refused it is gone. `AggregateFinalize`, `aggregate_exchange` and their tests are deleted; nothing in the crate referenced the exchange module. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): route aggregate state through the series repartitioner Replace the dedicated state-repartition node with the existing series repartition execution: state batches are routed by their canonical labels without being rebuilt as StateSeries, and each row slice is charged for the Arrow wrappers it newly allocates while the parent buffers stay charged once. Carry a producer's state charge with the chunk it hands over, so the exporter that packs rows across polls remains accountable for the batch it still holds instead of releasing it at handoff. Read a Final aggregate's value and tag columns positionally, which is the order the split's output schema is built in, and name them through the legacy expression only at the outer boundary; a bare sum over a metric whose value column is named after the whole expression no longer fails planning. Pack aggregate-state frames in one pass over positions and validity instead of scanning the contributing rows for every position. Cover the changed behaviour with execution tests for bare global and grouped sums over one- and two-value-field tables, for the handed-over state charge, and for the export lifecycle on drain and early drop. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bound tree depth on both sides and validate merge inputs The depth limit was decode-only and set to a number nobody had measured. A 129-deep tree could therefore be built, serialized and shipped, while the receiver refused it; and in CI the boundary test itself aborted with a stack overflow, because a 2 MiB thread was asked to build, serialize, decode and drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is not covered by a guard that only wraps `Deserialize`. Measured instead of guessed. A probe walked build, serialize, decode, validate and drop at increasing depth inside a named 2 MiB thread: 32 through 128 completed, 160 aborted. The cap is now 64, half the highest passing depth, and the same number is enforced in `NativeTree::try_new` and in the deserializer. Verified through the real constructor: a 64-node tree round-trips, and a 65-node tree is refused when it is built as well as when it is decoded, so sender and receiver accept the same set. The boundary test runs on a named thread, which is how a future failure will identify itself. The merge node accepted any single input. `with_exprs_and_inputs` rebuilt `AggregateFinalize` over whatever it was handed, on the reasoning that a rewritten input only changes where the state comes from. Nothing checked that it was state at all -- and until now nothing could produce a counterexample, because a values-producing producer could not be serialized. The serializer was standing in for a validation that belonged to the node. `with_exprs_and_inputs` now rejects an input whose schema is not `state_schema(binding.tags)`, so substituting a computed-values producer fails at the logical node with a message naming the expected schema, rather than depending on an encoding limitation. `AggregateFinalizeExec` already had the same check at physical planning; the serializer stays permissive. The three placement tests in `commutativity.rs` asserted the old refusal as their mechanism -- an ordinary fused node "fails to encode", a partial producer "cannot be pushed down". Those expectations were the encoding limitation wearing a test's clothes. They now assert the responsibilities that remain: an ordinary fused node round-trips through the encoder, a partial producer rides to the region while its finalize stays on the frontend, and a values producer handed to a merge is refused by the schema contract. The test-only `NPRQ` envelope is gone. It framed trees with its own version, a 1 MiB cap and its own rejections, none of which production had, so `native_tree_envelope_rejects_version_size_trailing_and_truncation` was proving a helper's protocol while the PR description claimed a production guarantee. Trees are tested through plain postcard, transport through the real `FusedSeries` interface. The redundant clone-and-revalidate in `FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already routes through `try_new`. Signed-off-by: discord9 <discord9@163.com> * style: let taplo order the workspace dependency entry `taplo format --check` runs over the whole workspace in CI, not over the two manifests I had checked. It sorts the `[workspace.dependencies]` table, so `postcard` belongs in alphabetical position inside the external-dependencies block rather than immediately after its comment. Local checking was narrower than the CI command, which is why this only surfaced there. Signed-off-by: discord9 <discord9@163.com> * style: place postcard in the sorted position the review asked for Inside the external-dependencies block, `taplo` was willing to leave the entry before the `axum` comment, because a comment line ends a reorder group. The review asked for the alphabetical position, which `taplo format --check` also accepts, so use it. Signed-off-by: discord9 <discord9@163.com> * docs(promql): name the real reason a multi-source name grouping declines The comments still said the fused node's declared columns cannot name `__name__`. Since #10106 a discovered source declares it, so a nameless selector's `sum by (__name__)` survives; what actually declines is a tree reading more than one *named* source, where the table name is the metric name and no source declares the tag. Same conditions, accurate reason. Signed-off-by: discord9 <discord9@163.com> * test(promql): find the topk producer by behaviour, not by node count `native_aggregate_topk_over_global_partitions` asserted that the whole plan holds exactly one fused node. Once native frames cross the region boundary a plan legitimately holds the frontend root and the region producers at the same time, so that count is no longer an invariant. The test now selects the segmented producer and the ranking node by what their plans do: at least one producer contains `segment` and none of them runs `topk`, and exactly one node across the fused and final captures ranks. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-11-04 21:25:41 +08:00
discord9	342e7879ed	feat(promql): admit label rewrites and decide canonical collection once (#10097) ... * fix(promql): let a closed chain keep its charge until the packer is done Four places released the frame reservation: the early-publish path handed it to the packer, but the queue drain and the end-of-input path freed it in place. Those two run while the packer still holds rows it has not turned into batches, so a series spanning several batches went unbilled once its last piece was handed over. Every emit path now hands the charge over, and the packed stream frees what it holds: what is left when it is polled and pending is the one identity it is turning into batches, and nothing survives its end or its drop. The identity guard is also widened: within one source every series shares its metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is what a range function, a math map and the timestamp shape do -- still maps distinct series to distinct labels. Only rewriting an ordinary label can make two equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame output therefore publish as their chains close instead of buffering a partition. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must Three defects in how this node decided a tree's execution layout. A match whose two operands derive from the same source still needs its match keys co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)` therefore fell through to single-partition execution and lost the parallelism the scan gave it. A raw input never received a matching router: the normalization branch returned it unchanged unless the whole tree was global, while the builder requires a direct matching router for any multi-partition matching-local input. A two-source raw query was rejected at planning. Nothing reconciled the partition counts of different sources either, though the builder requires one common positive count. A single count is now computed for the matching-local case and every router under it uses it, gathering when that count is one. The window restriction also ran before normalization decided the layout, so a tree about to be gathered could be refused for a limit that only applies to partitioned windows. Two tests changed with the behaviour rather than around it: hash routing does not promise which partition holds a given series, so that test compares results instead of placement while keeping its partition count, per-partition execution and operand-order assertions; and a test that pinned the old refusal of a multi-source frame now pins its acceptance. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): keep a raw source's scan partitions instead of gathering them `RawSeriesSourceExec` declared one partition and read every child partition inside `execute(0)`, so a cross-table node lost the scan's parallelism before its inputs reached it. It now declares the child's own count, executes exactly the partition it is asked for, and re-labels that partition's frames. That leaves `SeriesGather` with no consumer: it existed to interleave several partitions into one stream. The module is gone rather than kept as a hidden N-to-one path a later plan could quietly fall back on. Signed-off-by: discord9 <discord9@163.com> * fix(promql): publish a raw source's frames under the ordering it declared `pack_raw_series` derives the frame schema from the stream's own ordering, and `RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing the scan's stream through unchanged broke that agreement: a columnar scan promises key-then-time, so the region published key-then-time frames while the frontend still expected the schema it had built, and `MergeScan` refused them. The forwarding adapter restates the ordering this node declares. It is the weaker promise, so a scan that keeps a finer one still satisfies it, and every other stream property is forwarded unchanged. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw frame that names its labels by its key A region cannot hand out complete labels: it has no metric-name binding, so `prepare_raw_scan` refuses an explicit binding outright. What it hands out is the stored key plus the description a decoder needs, which is the normal input to this node rather than a case to refuse — every single-table selector is planned that way. The node's constructor refused exactly those frames, so any plan that inserted a router under a single-table selector failed before it ran: the matching policies this branch adds, and the one-partition normalization of an evaluated tree. The node publishes labels, because that is what it groups and closes identities by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the input is decoded once, at the unpack boundary this node already has, and an already-labelled stream passes through it untouched. The frame it declares is rebuilt from the input's own declaration instead of being compared against it, which keeps the check that a frame this node cannot re-declare is refused while letting input and output differ in how they name the same series. `frame_schema_with` becomes public for that rebuild, since it is the one place that decides what a declaration is. `FusedSeries` now describes the router's output rather than the scan's when it wraps a raw input, because the binding it hands `build` has to match the frame the router actually publishes. Tested through the node directly: a key-labelled input routes at all; matching puts both pieces of one label set in one owner and closes it once; and `RawPrimaryKey` still hashes the key rather than the decoded labels. Signed-off-by: discord9 <discord9@163.com> * test(promql): measure a routed pair by the width it routes to Four assertions still described the layout this branch replaced: a node with two inputs used to read every partition of both into one and answer one partition, and that gather is gone. The pair is now matched in the partitions it was scanned in, so the fused node answers as many partitions as its inputs route to. Each of these fixtures scans with four partitions, and on the base of this branch all five tests passed, so this is a behaviour change this branch made and did not re-verify rather than a defect: the value assertions behind these shape assertions pass unchanged, in both the native and the legacy comparison. The width is read from the routed children instead of being written as four, and the scan is found by walking down through the nodes that only pass its frames along, since a routed input is no longer a bare `SeriesScanExec`. Signed-off-by: discord9 <discord9@163.com> * fix(query): normalize a multi-partition series-frame root A query engine hands its plan to one stream, and for a plan with several partitions that means `CoalescePartitionsExec` - ordinary batch coalescing. A series frame cannot survive it: one stored series may be split across partitions, and a chunk's `is_last` closes that producer rather than the series, so interleaving two partitions' batches produces a stream whose decoder refuses it (`raw series key changed before final chunk`). Evaluated frames have the same shape of problem: their validator keeps one open chain, and a batch from another partition interrupts it. The engine's root is also a region's root, which is how a cross-table native statement reaches this: its per-region sub-plan is rooted at a partitioned `RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end, before the final sanity check, and it normalizes the root and nothing else - `FusedSeries` already inserts its own owner wherever a consumer needs one. A single-partition root is left alone: one producer is the whole answer. A key-labelled raw root is refused rather than coalesced, because the router checks that one stored key decodes to one label set and it needs the labels to do that. No production plan reaches that rejection today - the only producer of a key-labelled frame is the region's series scan, and every plan that executes it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish explicit labels - and the rejection is what keeps it that way. `schema_check` stays enabled: for the roots this rule accepts, the router re-declares its input's frame with the same content, label mode, decode description and ordering, so the schema it publishes is the one it was given. Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit an evaluated frame as a fused input The extension planner hands a child straight to the fused executor when that child already publishes a series frame, and until now it only recognised a raw one. A segment's output is an evaluated frame, so a statement planned as two segments could not be bound above the first one. The check it replaces asked for a raw frame; the property that matters is that the frame names its own series, which is what `labels() == Explicit` says - evaluated frames always declare it and carry no decode description. Everything else stays with `FusedSeriesExec::build`, which already refuses a mixed-content node, an evaluated input that is not contiguous, and a frame whose declaration does not match the binding it was given. Signed-off-by: discord9 <discord9@163.com> * feat(query): plan a global topk under an aggregate as two segments `sum(topk(1, m))` needs the whole vector for its ranking, which is a property of the tree rather than of the statement, and the routes that existed could only answer a whole-vector tree in one partition: the statement fell back to the existing planner. Splitting it lets the per-series prefix keep the scan's partitions while the ranking sees every candidate. The cut goes below the `topk`, not below the `sum`. A segment holding a `topk` reads across series, so it would gather its own input before evaluating the prefix - correct numbers, no parallelism, which is the thing the split is for. Below the `topk`, the prefix runs on the scan's partitions and the ranking and the fold above it run once, on the child's evaluated frame. Both segments are built from one compiled tree: the child is the prefix subtree, the parent is the original tree with that subtree replaced by an input bound to the child's frame. Nothing is re-parsed, so the grid, the matching rules and the bound scans are the ones the compiler produced. The parent needs its output columns without asking the existing planner to name a computed column after its expression, so `fused_output_tags` now derives the grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels through. It derives output *columns*, not a complete runtime label set: the metric name is bound separately from the tag columns, and a grouping by `__name__` keeps that name in the labels while contributing no column. `fused_root_schema` is that derivation, shared with the ordinary route. The segmenter is its own method, boxed at the call site. Inlined, its future made every frame of the planner's recursion large enough to overflow the stack on a deeply nested statement - `test_parse_or_operator` aborted before this. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the two-segment topk plan and what it keeps The statement's two segments do not land in one physical tree: the prefix has one scan for a child, so the categorizer classifies it commutative and pushes it into the region, while the aggregate above it stays at the frontend. The observer therefore records every plan a query engine wraps, not only the first, so the test can look at the region's own root. What the assertions pin, and why each one is worth its brittleness: - the answer is the global maximum, not a sum of per-partition winners; - the frontend segment holds the complete sum and the global topk over an evaluated input, with no instant evaluation left in it; - the region's segment holds the prefix and its output is an evaluated, explicitly labelled, contiguous frame; - that segment keeps more than one output partition, which is the whole point of cutting below the topk; - the region's root is the series-aware router, one output, and it reads that segment. Column names are not compared: this route names a computed column after its compiled root while the existing planner names it after the expression, which is older than this split and not what this test is about. Signed-off-by: discord9 <discord9@163.com> * feat(promql): share the input-locality classification A tree's distribution requirement is one property, and two layers now ask it. The physical planner already decided with `input_locality` what a tree needs from its inputs; the logical planner has to ask the same question when it splits a statement, or the two can disagree about what a subtree requires. `InputLocality` and `input_locality` are therefore public within the workspace. The classification describes the external inputs a tree needs - not its output distribution, and not which neighbouring operators may share a node - so a caller that reads it as either of those will cut where it should not. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement where a gather would swallow its prefixes The previous commit planned `sum(topk(k, prefix))` by matching that shape. A planner that grows one matcher per query shape grows one per benchmark entry, and the shape is not what decides where a statement divides: the tree already says what each subtree needs from its inputs, and `input_locality` answers it. So a segment is now a maximal subtree with one distribution requirement. Inside a tree that must gather, any child that (a) can cross a boundary - it produces evaluated samples rather than raw samples, aggregate state or a scalar - and (b) does not itself need a gather, becomes a segment of its own and keeps the partitions it was scanned with. Nothing else is cut, and a tree with nothing to cut keeps the plan it had. Every segment rebinds its own inputs: the scans it reads directly first, in the order it first names them, then the segments cut out of it, because `FusedSeries` requires dense source indices over its children. `fused_output_tags` derives the parent's columns from those inputs rather than from the statement's first selector. Three gates keep this from answering what the routes before it would not. Each one is a failure this change produced before it was added: - the existing planner must still be able to plan the statement. Deriving tags locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` - an empty range, which the existing planner rejects - was answered instead of failing; - every segment's tree must construct a `NativeExecutionState`, because nothing below can fall back once physical planning starts, and a grouping by a value field is a shape the runtime refuses; - the tree must contain a prefix that a boundary could carry, so a whole-vector tree with no per-series work inside it is not split for nothing. The scope is one selector. A statement over two tables binds each scan as a `RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in a region, so its prefix would not gain the partitions the cut is for; those statements keep the plan they had, which the acceptance test pins. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the segment boundaries and their scope One statement per shape the cut can meet, asserting what the rule says rather than what one query happens to produce: a per-series tree stays one node, a matching tree routes instead of gathering, a global fold over per-series work cuts that work out, a constant stays embedded because a boundary cannot carry one, and a statement over two tables keeps the plan it had because a segment over `RawSeriesSource` cannot be placed in a region. Answers are compared as sets of rows with the value column renamed. Three things differ between the routes and none of them is the answer: this one names a value column after its compiled root while the existing planner names it after the expression, the two emit columns in a different order, and this one is partitioned so its rows arrive by partition. Comparing them as what they are leaves the values as the only thing an assertion can fail on. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep a selected segment whole A segment is chosen because its subtree needs one distribution; cutting inside it again contradicts the choice. The inner builder was still applying the cut predicate, and that predicate is "exportable and not `SinglePartition`" - which every node inside a per-series subtree satisfies, since a per-series tree has no `SinglePartition` node in it to begin with. `sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one segment, then `abs` was cut out of that segment, leaving a middle segment whose child is a node rather than a scan. That middle segment cannot be placed in a region, so it answered one partition while the scan had several - the prefix lost exactly the parallelism the cut is for. The inner builder now collects the segment's `Input` leaves and nothing else. Signed-off-by: discord9 <discord9@163.com> * refactor(query): bind a segment to its own scan A cut segment carries one source's identity: its scan, its metric name, its time index, and its labels. Building it from the statement's first selector only happens to be the same thing while a statement has one selector. `build_inner_segment` takes the two lists the planner already has - each occurrence's filtered scan, and each occurrence's metric, time index and tags - and binds the segment's single source from them. The parent keeps the wrapped children and the statement-level binding it has now, because a source that reaches the parent directly still has to cross a region as a raw frame. A segment that collects more than one source is declined. `MatchingLocal` satisfies the cut predicate while reading two sources, and one segment cannot carry both sources' identity. No behaviour changes: the entry is still one selector, so no scan is wrapped and the statement-level values are already that selector's. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement that reads more than one table A whole-vector statement over two sources is now cut into one per-series prefix segment per source, with the fold above them in the frontend. Each segment binds its own source's scan, metric, time index and labels, so it can be placed in the region that holds that source, and each prefix keeps the scan partitions it was read with. Three things had to change for the existing segmenter to reach that shape: The cut sat inside `!per_partition && !multi_input`, so a statement over two metric tables - which is `multi_input` - never reached it. The cut now sits under `!per_partition` alone. The aggregate split keeps its own conditions, including `!multi_input`, so a multi-input tree is still not split that way. The single-selector gate was doing two jobs. A statement naming several selectors declines unless those selectors read more than one table: the count alone says nothing about whether the tree has more than one input to place, and the shapes it would otherwise admit - one metric named twice under a fold, such as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers. `cross_table` rather than `multi_input` decides this, because two Mito tables carry no metric logical id and are still two sources. A segment that collects more than one source is declined. A matching subtree over two sources is a cut candidate, and one segment cannot carry both sources' identity, so `sum(topk(1, m / n))` keeps the plan it had. The segmented parent reports the value column the compiled tree names, not the one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where the existing plan reports an expression-shaped name. The values are identical and both names are pinned in the test, so a change to either rule shows up there. Restoring the existing name needs a rename above the parent - the node's own schema must keep the compiled names its output columns are classified by - and is left to a follow-up. Signed-off-by: discord9 <discord9@163.com> * test(query): pin that two sources keep their own scan bounds A segmented statement binds each prefix to its own source's scan, and A1 made that binding: `bare_scans` is derived from the prepared scans after each selector's time filter has been pushed into its own `TableScan`. Nothing tested that the two segments actually read `their` bounds rather than one window applied twice - the existing two-source case has no offset, so both sides read the same range and a crossed binding would answer the same numbers. `sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only in the right side's window, so the two queries must disagree: 20/1 and 30/2 against 20/2 and 30/3. The test asserts they do, which is what fails if both segments were given the left scan's bounds. The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints only the peers: the debug form carries the table each segment reads and the embedded `FusedSeries` segment. An in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence for segments is measured in `native_aggregate_test.rs::check_two_source_segregation` instead. Signed-off-by: discord9 <discord9@163.com> * fix(query): decline a multi-source tree that groups by the metric name The gate that lets a multi-source statement reach the segmenter is `selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`. Two ordinary Mito tables differ in table name and carry no metric logical id, so they are `cross_table && !multi_input`: the guard did not apply and the statement reached the segmenter. `fused_output_tags` keeps a grouping by `__name__` out of the declared columns on purpose - the name travels in the chunk's labels, not as a tag column - so `sum by (__name__)` over two sources derives no tag column while the runtime holds two groups that differ only by metric name. Both are then reported under one label set, as two rows a reader cannot tell apart. That is the failure the metric-logical-table test already documents; it could not catch this path because its tables make `multi_input` true. The check is scoped to a tree that reads more than one source. One source folds one metric, so its groups carry the same name and naming that name or dropping it cannot merge two of them. Signed-off-by: discord9 <discord9@163.com> * test(query): cut a range function on each side of a division The two-source cases so far cut an `Instant` prefix: one sample per grid point. A window function's prefix computes its window inside the region over that source's own samples instead, so the segment carries a `Window` rather than an `Instant` and nothing tested that such a prefix is still a cut candidate. `sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then 4/3. Two different step values are what let a shared window bound change an answer, which is asserted before the result is compared. The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset case: an in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence lives in `native_aggregate_test.rs`. This is the shape of the throttling query in the benchmark corpus: a window under a fold, on each side of a division. Signed-off-by: discord9 <discord9@163.com> * test(query): plan the benchmark's throttling and namespace-ratio shapes Every two-source case so far divides two ungrouped scalars. The corpus queries that motivate this work group by a label on each side, so each segment reports several groups and the parent matches the two sides by that label. Nothing tested that path. `container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total` is the corpus query verbatim: two metric logical tables on one physical table, `sum(increase(...)) by (container)` on each side. It plans as one frontend `FusedSeries` over two `MergeScan` children, each holding a segment bound to its own metric, and reports the same labelled rows as the existing plan. The namespace-CPU-usage-over-requests shape differs from the corpus text only in the metric names, because the corpus names carry recording-rule colons that a table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form and is asserted as that shape rather than as the query verbatim. Both tests assert the native route answered, so a decline fails here instead of passing on the legacy rows. The value column still differs from the existing plan's - the same known gap recorded for the ungrouped case - and both names are pinned. The fixture helper records the three setup traps the hard way: `INSERT` needs explicit column lists because the physical table's column order differs from the logical table's, SQL keywords used as tags need quoting, and no sample may sit at timestamp zero, which both the scan's lower bound and PromQL staleness drop. Signed-off-by: discord9 <discord9@163.com> * test(query): record which native route every corpus query takes The corpus is 29 queries and nothing said which of them the native route answers, or by which of its plans. One parameterised test now runs all of them over generated fixtures and records, per query, the route and the numbers. Four routes, and the frontend tree alone cannot tell them apart: - `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one per source; the prefix work happens in the regions. - `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region `FusedSeries` that reports `partial state:`, so what crosses the boundary is aggregate state. - `region-local` - the frontend holds no native node at all: a per-series tree is commutative, so the whole `FusedSeries` is pushed into the region. A classifier that only looks at the frontend calls these legacy, which is what happened before this test existed. - `legacy` - no native node on either side. All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two `region-local`, none legacy. The two `region-local` ones are the corpus queries that fold nothing - a bare selector and a `rate` - which is also why pushing the whole tree into the region costs nothing. The test asserts only that the existing plan answers each query with rows, and that a native route's labelled rows agree to within `1e-10` relative. It records the route rather than asserting it, because which route a shape takes is a property of the planner that this test exists to report, not to legislate. Ten queries differ from the existing plan in the last bits; the largest relative difference is 2.9e-16, which is summation order, not arithmetic. The table prints that figure per query so a future equality failure can be told apart from rounding without re-measuring. Every fixture here is generated: explicit `INSERT` column lists, quoted SQL keywords, and no sample at timestamp zero. They establish the plans and the values; they say nothing about speed, memory, or how many partitions actually produced rows. Signed-off-by: discord9 <discord9@163.com> * test(query): record how many partitions each corpus query's nodes used The route matrix said which plan answers each query. It did not say whether that plan produced anything, so a node that ran and emitted nothing looked the same as one that worked. `output_values_per_partition` reads `native.root.output_values` off a `FusedSeriesExec` and returns the counts per partition, and the matrix prints them as `partitions used`. `assert_four_native_partitions` keeps its own assertions and now builds on the same helper. What the column does not establish, and says so where it is printed: these fixtures hold a handful of series each, so a scan puts them all in one partition - every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the plan. The one place a fixture is built to fill the scan's partitions is `native_aggregate_test.rs::check_two_source_segregation`, which asserts all four produce values for both segments of a two-source statement. This matrix records a single assertion instead: at least one partition produced values, so a node that ran and emitted nothing fails. Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise divisions. Their region side has no `FusedSeriesExec` at all, because both scans are matched inside one region: that is the route, not a missing measurement, and the table says `no metric` rather than implying zero. Signed-off-by: discord9 <discord9@163.com> * test(query): record the new rule in the explain stages `TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering `SeriesFrameOutput` adds one line to that listing whether or not the rule changes the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary tables, whose schemas carry no series frame declaration, so the rule returns the plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the existing lines say for the same reason. Regenerated by the sqlness runner; the diff is those three lines and nothing else. Signed-off-by: discord9 <discord9@163.com> * test(query): make the corpus matrix prove what it claims Three gaps, all in the test the "every corpus query answers natively" claim rests on. The partition helper returned early on a plan that reported no `native.root.output_values`, so a producer that registered nothing passed on the declared partition count alone. It now expects the metric: the point of the assertion is that every partition executed, and an absent counter says nothing about that. `segmented` was decided by the frontend tree containing a `MergeScan`, which also matches a frontend native node consuming raw frames from its regions - a different placement that says nothing about a prefix running in a region. The merge scan's `Debug` carries the plan its region runs, so the check now reads that for a segment; the raw case gets its own `frontend-raw` route. Four queries move there: L05, L06, L08 and R03. The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been probed in a throwaway run. They are entries now, with the join labels Q08 and Q09 match on and real `le` buckets for Q10's histogram. A fallback is no longer a route to record: a case that reaches the existing plan fails, because that is the regression this table exists to catch. Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29. Signed-off-by: discord9 <discord9@163.com> * refactor(query): let the compiler decide which calls this route answers A second list of function names stood beside the compiler's own capability check, and the two disagreed. `admitted_selector` named five calls; the compiler folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same call meant two different things depending on what wrapped it. The compiler already answers this question: `compile_call` refuses a name it has no kernel for with `NotImplemented`, and the planner turns that into a decline. With the list gone there is one place that decides, and a call the compiler folds reaches the route whether or not it is wrapped in an aggregate. `sort` and `absent` read across a vector rather than one series at a time, which is why the old list named them out. They compile, so they are planned; the tests that asserted the refusal now assert the admission. Only the shapes the compiler genuinely cannot express keep the existing plan - a subquery evaluates its body on a grid of its own, which this route does not offer. Signed-off-by: discord9 <discord9@163.com> * refactor(query): admit an operand by its shape, not by call name An operand of a hybrid binary was admitted only if it was a selector under one of five named calls. The compiler folds many more than those five, so `ceil(m)`, `irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under an aggregate was admitted - the name list disagreed with the compiler about what this route can answer. What the list was really protecting is a layout assumption, and the shape states it directly: `try_plan_native_operand` hands back a selector's planner context, and the hybrid caller reads field columns and tags off that context, so an operand whose output layout is not its selector's cannot be one. `sum(A) / B` is refused for that reason, not because `sum` is a name. `single_selector_spine` states it on the expression: a selector, parentheses, unary, a call carrying exactly one vector-or-matrix argument, or a binary whose other side is a compile-time constant. Calls are followed by the type of their arguments rather than by `constant_value`, because a call's string parameters are constants that `constant_value` does not fold and would otherwise look like data. Which calls this route answers is now the compiler's answer alone: a name it has no kernel for comes back as `NotImplemented`, which the planner turns into a decline. Signed-off-by: discord9 <discord9@163.com> * feat(promql): mix raw and evaluated inputs in one fused tree `build` refused a tree whose children were not all raw or all evaluated, but nothing below that point assumed uniformity: `execute` picks the decoder from each source's own schema, `FusedFeed` carries the source index, and `collect_source_schemas` already validates a repeated source against its own grid. A frame stream has one format per stream, not one per consumer, so one segment can decode the raw samples of a rate and compare the result against a vector a previous stage already evaluated. Remove the refusal and make the remaining per-input decisions per source: `evaluated_global` now asks about the source it orders, and `window_sources` keeps only the evaluated sources, since a raw source is answered by the raw path it always used. Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y` arrives evaluated on the statement grid. The test asserts all ten (host, timestamp) rows and fails on the removed refusal if the refusal is restored. Signed-off-by: discord9 <discord9@163.com> * feat(promql): bind a segment to every source its subtree reads `cut_as_segment` accepts any exportable subtree that is not `SinglePartition`, so a matching subtree over two sources - `m / n` - is a legal cut point. `build_inner_segment` then required exactly one source and returned `None` for it, and `segment_tree` propagates that `None` as "the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back to the legacy path even though its cut was legal. Build the segment from every source the subtree reads. Each child gets the `RawSeriesSource` boundary already used for separate metric tables, which carries that source's own metric, tags and fields across the boundary and makes the child placeable; a segment over two tables cannot sit inside either region, so it runs above them. The single-source case keeps its existing direct-scan path unchanged, which is what lets a one-table segment still be pushed into its region. Field names come from `tree_source_fields` on the rewritten tree, whose source indexes are already dense in the order the children are built. That helper also validates a repeated source against its own schema and refuses a non-dense index set, which a second traversal would have to re-derive. It becomes `pub` and is re-exported for the query crate to call. Take the segment's time column from its own first source rather than from the statement's first: a cut whose sources are `[1, 2]` would otherwise name the time column of a table it does not read. Cover the construction with a planner test and with a real-Mito integration test that asserts both the exact values and the route: one segment over the two tables, each child reading its own table through its own raw series source. Without the route assertion a silent fallback would pass on equal values. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a matching tree at the configured width A matching tree's width came from its inputs: `matching_partitions` took the selected source's count, or the maximum of every input's. In standalone a cross-table statement arrives as one `MergeScanExec` per table with one partition each, so `m / n` matched on a single worker while the scans under it ran in parallel. The engine's `execution.target_partitions` was reachable - `plan_fused_series` already uses it for `series_scan` - but never reached this decision. Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which resolves a matching tree's width as `target_partitions.max(1)` for both the same-source and the multi-input case. `to_execution_plan` keeps its signature and its input-derived width, so a caller that has no configured parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do not consult the value at all: a per-series tree runs on whatever partitions it was scanned with, and asking for a router there would only add a shuffle. Forward the value through `FusedSeriesExec` because `with_new_children` rebuilds from the node's own children, which are the ones below the router; rebuilding without it would resolve the width again from the pre-router count and silently fall back to one. Verified on a real standalone: the two-table segment now reports four partitions with a `SeriesRepartitionExec` under each one-partition region scan, where it reported one before, and the rows are identical. Signed-off-by: discord9 <discord9@163.com> * test(promql): declare a keyed test frame's ordering, and order its keys `plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`, which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it returns packs what `RawScript::ordering` reports, which is now the key-then-time order this helper models. The node reads the ordering off the *plan* schema, so the two disagreeing meant the router scheduled these fixtures as `Contiguous` while unpacking validated them under the stricter ordered contract. Build the schema through `frame_schema_with` with the ordering the stream actually produces, so the plan and the runtime frames state the same contract. That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with both sides claiming `contiguous`, the ordered validation never ran. Running it showed the matching fixture feeding key `b` before `a` within one partition, which its declared ordering forbids. Reorder the partition; the two pieces of `host="a"` still arrive in different partitions, which is what the test is about. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): declare the key order a two-phase raw scan actually keeps `SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a `TwoPhase` scan that is not reading columnar artifacts. The data that path produces is ordered by primary key: - candidates are merged and sorted ascending by the encoded primary-key column, then decoded to ids by walking that ordered stream without reordering; - partition assignment is a deterministic TSID-to-partition map that appends in arrival order, so one series never spans two partitions; - each assignment is drained completely before the next is taken, rather than emitted in completion order; - the raw adapter's predicate buffering preserves input order and cannot let a later key overtake a held one, and two-phase records always report `ends_package: false`, so it never engages the package-tail machinery that legacy relies on. Declaring the weaker order costs two things that are available: the frame codec's `SequenceValidator` never ran its increasing-key check, and a repartition owner could not use a key watermark to release an identity before another input ended. It also contradicts the scanner's own documented contract, which states the output is always ordered by (primary key, time index) within a partition. Report the order the path actually keeps, and reword the two ordering errors that said "legacy" now that both modes can reach them. Cover it with a test that forces two-phase mode and the non-prepared fallback, asserts which of the two it ran, and then checks per-partition key monotonicity, per-series timestamp monotonicity, single-partition ownership, and the ordering the scan reports. The fixture writes three rounds, uses two output partitions and a residual predicate, and keeps one series above a batch so the multi-batch path is exercised. The three existing two-phase raw tests asserted the old declaration. They check that a series stays in one run and never recurs non-adjacently, which the stronger order still guarantees; only their expected declaration and their names said "contiguous", so those change and the substantive assertions stay. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw match only by a key the computation keeps `same_source` answers "this subtree reads one source", and a `MatchingLocal` classification was taking that answer as licence to insert the router below the fused computations, hashing the labels as they are at the raw stage. Those are different facts. A math map drops the metric name and the `__type__`/`__unit__` metadata, `timestamp` does the same, and every range function drops them except `last_over_time` and `first_over_time` — so `on(__name__)` over `abs(...)` routes by a key the very next node deletes. Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side routes `x` and `y` to different owners; both become the empty key after `abs`, nothing re-partitions between the map and the match, and the pair is silently lost. Decide preservation separately from source identity. A node keeps the key when it cannot remove one of the labels the key is built from; a `Window` consults `RangeFunction::drops_metric_name` rather than assuming, so the two functions that keep the name still route early. Anything not proven to keep the key — a label rewrite, a histogram map, an unmodelled node — does not establish preservation, and the tree is classified `SinglePartition`. That is not a fallback that trades parallelism for correctness. When the computation removes the labels the key is built from, every series on one side reduces to the same empty key, so every left series matches every right one: the match is whole-vector by construction and has no partitioned form that preserves its result. One partition is the only placement that answers it. Where the key does survive, the match stays partitioned. Signed-off-by: discord9 <discord9@163.com> * fix(query): describe an admitted operand by what it produces `try_plan_native_operand` handed the binary matcher the context saved during scanning, which describes the operand's *input* layout. An admitted call can change that layout: `histogram_quantile` folds a classic histogram and drops `le`, so the scan context still names a tag column the operand no longer produces, and default matching then builds a join key over a column that is not there. Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)` declines whole-tree planning (two selectors over one table) and is then admitted operand by operand; each operand drops `le`, and planning fails building the join with `FieldNotFound { field: "le" }` against a schema of `timestamp, field_0, host`. Publish only the tag and field columns that the operand's own plan schema actually contains, and drop `use_tsid`, which the produced output does not carry. The check is against the schema the node returns rather than against a list of functions that happen to shrink it, so a call added later cannot reintroduce the gap by not being on a list. Covered by a classic-histogram fixture with `host` and `le` and a float bucket field, asserting each operand's context reports `["host"]` and that the planned join key omits `le`. Reverting the change fails that test with the `FieldNotFound` above. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the partitioned duplicate-group answer, with the gap named `native_hybrid_binary_05_06_08` asserted that the right side's two `job="other"` series are refused as a duplicate match group. On the partitioned route they are not: the check is gated on `lhs.presence && rhs.presence` computed from the partition's own slices, so the partition holding `other` sees no left-hand data and filters every point out. Pin what the route actually answers, and state the three things that constrain the fix, because each has already been tried and rejected here: - the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a Prometheus-generated case where the left is absent at one point and the right carries three samples of one group there, and the answer is a value with no error; - the scan cannot run unconditionally, for the same reason; - the match cannot be collapsed to one partition to make both operands whole, because that gives up the parallelism this route exists for. What remains is to let the verdict see whole-operand presence while the partitions keep answering — a deferred check rather than a different placement. This test now makes any change to the current answer visible instead of silent. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a nested match instead of gathering it A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` — passes the logical admission check, because `independent_scan_pair` recurses through `scan_operand`. The physical locality classification does not go that far: it recognizes a match only when both operands read one source each, so `same_source` fails on an operand that is itself a match and the whole tree is classified `SinglePartition`. The fused computation then ran on one partition while the scans under it stayed parallel, which the existing planner does not do: it builds ordinary equijoins that DataFusion can execute as partitioned hash joins. Admission and placement are different questions, so they now get different predicates. The tree is segmented when it is cross-table, not multi-input, classified `SinglePartition`, and its root is a vector match with a nested vector match directly under it. The inner match becomes a segment and the outer one repartitions the evaluated operands by its own key, which is what the segmenter already does for any exportable evaluated prefix; a nested match whose two levels use different keys is routed twice, once per key, rather than pretending one routing serves both. The predicate stays narrow on purpose. Three-selector trees over distinct metric logical ids are `multi_input` and keep their existing construction, including the parent binding that construction produces. A same-table nested match does not enter this branch either, so its prior admission is unchanged. The plan test's collector walks each fused node's native tree in expression-child order and resolves each input's `source_index` against that node's inputs, so a segmented plan is checked for the same property as before — every selector occurrence bound exactly once, in expression order — rather than being checked more weakly. It panics on a shape it does not model instead of returning an empty list. Signed-off-by: discord9 <discord9@163.com> * fix(promql): prove a raw match key survives the computation `preserves_matching_key` decides whether a subtree keeps the labels a vector match routes by. Two arms answered yes without looking at the field that decides it. A binary with a scalar operand recursed into the vector side alone, but arithmetic and a `bool` comparison drop the metric name while a filtering comparison keeps it — the distinction execution already makes. And an `Instant` recursed unconditionally, while `timestamp(x)` compiles to `InstantValue::Timestamp`, which drops the name too; the check written for the separate `Timestamp` node never covered that shape. Both now read the field that decides: the operator together with `return_bool`, and `InstantValue`. The labels checked are the whole computed set — name, type and unit — because `drop_metric_name` removes all three and default matching does not ignore the last two by itself. Without this, `(x * 2) + on(__name__) (y * 2)` was classified matching-local, the router hashed the raw names before the maps dropped them, and a valid pair could land in different partitions with nothing re-partitioning afterwards. The regression asserts the value is 10 and that the tree is `SinglePartition`, computing the destinations the router would have used (`x0` to partition 0, `y0` to partition 1 of two). With the old arm restored the value assertion fails with `left: [10.0], right: []`. Signed-off-by: discord9 <discord9@163.com> * feat(query): find a nested match through the wrappers around it Segmentation was gated on the root itself being a vector match with a nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never entered: the wrapper or the scalar broadcast sat where the match was expected. Meanwhile the segmenter already walks children and leaves a wrapper in the parent while cutting what it can carry, so no per-wrapper case is needed — the search just has to look through them. Replacing the root check with `carries_prefix` was too wide: almost any multi-source tree contains something a boundary could carry, so ordinary shapes like `some_metric unless on(tag_0) other_metric` were cut for no gain and their inputs stopped being raw sources. So keep requiring a nested vector match, and search for it anywhere in the tree rather than at the root. Wrappers and scalar broadcasts are transparent because the walk visits every child; the nested match itself is still required, which is what keeps the plain multi-source shapes on the plan they had. `cross_table`, `!multi_input` and the `SinglePartition` locality requirement stay, as before: they are what keeps distinct metric logical ids and the same-table shapes out of this branch. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): let a router publish an identity whose chain is complete `SeriesRepartitionExec` published an identity only once every input had contributed it or ended, because an identity may be split across inputs. A per-series region scan assigns each identity to one partition, so on that shape the rule holds every identity until the other partitions reach EOF: a region's whole raw selection is retained before the first series leaves. The node now carries an `IdentityCompletion`, defaulting to the existing merge. `CompleteDisjoint` publishes an identity as soon as its chain closes and keeps the published identities charged in the pool, so a second chain for the same identity fails the execution instead of emitting a partial series as final. The declaration is the producer's, not something a consumer infers from a node's type: publishing early against a producer that does not own its identities would be wrong, so the conservative mode stays the default and every existing constructor keeps it. Signed-off-by: discord9 <discord9@163.com> * fix(query): choose a series-frame root's router from declared provenance `SeriesFrameOutput` normalized every multi-partition frame root through a router that merges identities, which is wrong for a root whose identities are already disjoint: it waits for every scan partition to end before the first series leaves. The rule now selects `CompleteDisjoint` only for a root whose producer declared that property. A raw root qualifies when `RawSeriesSourceExec` was built through `to_per_series_execution_plan` - which also checks the child is a `SeriesScanExec`, so the type alone is not taken as proof - and a direct `SeriesRepartitionExec` root qualifies because it merged the identities itself. Everything else, including a fused frame root, keeps the merging router: a native label transformation can map two source series onto one output label set, so its output identities are not exclusive. The rejection of a key-only raw frame is unchanged and its reason is restated: coalescing cannot produce such a frame, and a router needs explicit labels to check that one key has one binding. Signed-off-by: discord9 <discord9@163.com> * fix(promql): report the labels `absent` names, not the table's columns `fused_output_tags` answered `None` for `absent`, so the caller fell back to the existing planner's column names. Those are the table's tag columns, which is not what `absent` reports: `absent(m{host="x"})` carries the labels its matchers name, and the compiled tree already knows them in `BoundVectorOp::Absent { labels }`. The label set is now read from that field: the named labels are kept in the input's column order, so `absent(native_shapes{host="missing"})` over a table with `host` and `le` reports `host` rather than `ts, val, host, le`. A label the matchers name that is not an input column still answers `None`, because a frame has no column to carry it; `count_values` and `histogram` are unchanged. Caught by the native-versus-legacy value comparison: the extra column made `absent` differ from the existing plan while every value matched. Signed-off-by: discord9 <discord9@163.com> * fix(promql): describe a frame root's emission from what it does `FusedSeriesExec` declared `EmissionType::Final` for a frame output, but `poll_output` hands a frame's evaluated chunk over while its input is still arriving, so the declaration overstated the node to the optimizer. Only a partial aggregate's state is emitted after EOF; every other output keeps its input's emission type. Signed-off-by: discord9 <discord9@163.com> * test(promql): compare the shapes the native route now admits with the existing plan Removing the shapes a name list used to refuse put single-selector `sort` and `sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route. Nothing compared their answers with the existing plan, so the widening was unverified. The new case runs each shape both ways over one metric table. `sort`, `sort_desc` and both `absent` cases compare the whole printed result, which covers column names and values together. `histogram_quantile` compares values row by row and asserts the one known difference instead of hiding it: a segmented statement names its value column from the compiled tree (`val`) where the existing plan names it after the expression that produced it. That case is given a window long enough to produce a row, so it cannot pass by comparing two empty results. The `absent` cases caught a real difference: the native route reported the table's tag columns rather than the labels the matchers name, which is fixed in the preceding commit. Signed-off-by: discord9 <discord9@163.com> * fix(query): publish a fused root's identities early when the tree keeps them The root rule publishes an identity as soon as its chain closes only for a root whose producer declared that each identity lives in one partition, and it knew two such producers: a per-series raw scan, and a router that had already merged its identities. A fused root was always treated as needing a merge, so a region whose segment is a plain instant selection over one per-series scan still held every identity until the other scan partitions reached EOF - `sum(topk(1, m))` is that shape, and its segment is already multi-partition with no label rewrite. `FusedSeriesExec` now answers whether its output identities are disjoint, and the rule asks it. The answer is narrow on purpose: one input, and a tree of nothing but inputs and sample instant selections, over a per-series scan or another fused node that already answered yes. Anything that computes labels is excluded, because a timestamp or a pointwise map can map two source series onto one output identity, and publishing early would then hand over one identity's pieces as if they were different ones. A vector-to-vector binary reads across series and is excluded too. Signed-off-by: discord9 <discord9@163.com> * fix(query): segment a computed matching key's operands instead of gathering them A vector match whose operands compute away one of the matched labels was classified `SinglePartition` and executed on one partition. That is correct for the bare tree - the raw names cannot route a match whose operands drop them - but it also gave up the parallelism the operands still have: `(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so the computed sides can be routed by what they actually carry. The planner now cuts those operands into segments of their own and lets the parent route their evaluated outputs by the unchanged match key. The key itself is not rewritten: `group_key` reads the labels present at that point, so `On(__name__, host)` degrades to the `host` a computed operand still has. Which shapes take this route is decided by one predicate shared with `input_locality`, so the admission rule cannot drift from the locality judgement: a matching site whose two operands are each proven single-source computations, at least one of which does not preserve the key. A failed preservation proof on its own is not enough - it can also mean the proof gave up, as with a parameter that is a compound expression or an operand that is itself a vector match - and the shared doc says so. The same table read twice is admitted for this shape only. The rule that declines several selectors on one table exists to keep folds such as `sum(m) / m` off this route, not because one table cannot be read twice; the compiler already gives each occurrence its own source index, and each cut gets its own input. Distinct partitions are what the two operands need, and they get them. The regression executes both the same-table and the cross-table shape over a four-host fixture and asserts that at least two output partitions produce rows, so the result is not silently produced by one of them. Signed-off-by: discord9 <discord9@163.com> * fix(query): take the early-publication guarantee from the scan's binding The root rule publishes an identity as soon as its chain closes only when the producer can promise that each identity lives in one partition, and the fused answer to that question asked whether its input was a `SeriesScanExec`. The node's type is not that promise: a scan binds a set of logical tags and decodes those into the series' labels, and `try_to_named_series_scan` requires the caller to pass complete tag names but does not check it. A binding that names only some primary-key columns maps two stored series onto one output identity, and early publication would hand one identity's pieces over as different ones. The answer now reads the binding: the label source has to be `PrimaryKey`, and every primary-key column has to be among the bound tags. A source that carries its labels explicitly is not this shape and answers no, as before. Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is false, full coverage is true, an explicit source is false - and by tests for the predicate's two other parts: a tree that computes labels and a tree that is not one instant selection over one input both answer no, and a gated execution shows a completed identity leaving a fused node while the other input is still held. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop the checks the types and decoders already guarantee Two re-checks ran inside the repartition node over facts its inputs had already established, and both cost real work on every query. A raw merge scanned each chunk's timestamps for strict increase, but `RawTimestamps` cannot hold an unordered buffer: its field is private, its only constructor validates, and the accessors are read-only or order-preserving, so no safe caller can build the value this check defends against. The check is gone; the identity, label and shape checks around it stay, because those compare different producers' pieces rather than one chunk's own timestamps. An owner rebuilt each chunk's identity to prove a chain does not change it, but the decoder that hands the chain over already refuses that: `SequenceValidator` holds one open key and rejects a key or metric change before the chain closes, `EvaluatedSequenceValidator` does the same for labels, and the input task that assembles chains only reads from those decoders. What stays is the identity the owner needs anyway to pick a destination, now built by the named functions rather than inline. Where a schema was needed, the node built an empty stream of its own, wrapped it in a packer and read the packer's schema back. The frame module already answers that question from the declaration, so the fake stream is deleted along with its half of the roundtrip. Signed-off-by: discord9 <discord9@163.com> * fix(promql): recursively segment native matching plans Signed-off-by: discord9 <discord9@163.com> * fix(promql): derive native schemas without legacy probes Signed-off-by: discord9 <discord9@163.com> * fix(promql): recognize complete metric scan identities Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23) * fix(promql): release completed disjoint identity history Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c) * test(promql): assert native aggregate schemas independently Signed-off-by: discord9 <discord9@163.com> * fix(promql): identify internal metric keys by reserved id Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep parallel sources and derive group match labels Signed-off-by: discord9 <discord9@163.com> * test(promql): observe the frontend fused root in scan-parallelism checks Signed-off-by: discord9 <discord9@163.com> * fix(promql): release duplicate evidence immediately on abort Signed-off-by: discord9 <discord9@163.com> * test(promql): preserve operand identity across matching repartition Signed-off-by: discord9 <discord9@163.com> * fix(promql): charge the final duplicate verdict until it is reported Signed-off-by: discord9 <discord9@163.com> * test(promql): observe native scans across frontend and region plans Signed-off-by: discord9 <discord9@163.com> * chore(promql): drop dead test helper and clippy lints Signed-off-by: discord9 <discord9@163.com> * fix(promql): select limitk by canonical label order Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep data-free subtrees in the parent segment Signed-off-by: discord9 <discord9@163.com> * test(promql): prove the duplicate error crosses matching partitions Signed-off-by: discord9 <discord9@163.com> * test(promql): drop debug prints from the segmentation test Signed-off-by: discord9 <discord9@163.com> * revert(promql): keep limitk's Prometheus input-order semantics Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit label rewrites on the native route The native route declined label_replace and label_join outright, because the runtime refuses two series that a rewrite gives one canonical label set where the existing plan answered them. That refusal is what Prometheus does, so the existing answer was the wrong one, and on the current parent the refusal already reaches a rewrite that folds series together: the collapse errors at the root, under sum and under topk. Admit both functions: only the first argument is a vector expression and contributes selectors, and each function's arity is checked before its selectors are collected. Signed-off-by: discord9 <discord9@163.com> * feat(promql): derive canonical ownership and insert the merge consumers need Stacked on the frame-segment route so the ownership this pass proves is the ownership the physical plan actually gives it: a merge is inserted only where its input is co-located by canonical key, which the routing boundary establishes. The pass answers one series per canonical label set at the consumers that read series whole, and leaves the point-wise, kernel-filed and aggregate paths alone - their own duplicate checks already refuse two values of one field at one point. (cherry picked from commit d0bbfc6893) Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the label rewrite shapes the native route now admits The rewrite test still asserted that every label_replace and label_join keeps the existing plan, which was the admission this branch removed. Assert what the route does now: the shape is admitted, its rewritten label is declared, and the plan differs from the one built with the flag off. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): decide collection once, in the predicate both readers use Runtime construction carried its own answer to the question "may this node's output hold several fragments of one label set?", computed inside a sixteen-arm match and returned alongside every built node. The insertion pass asked a second implementation of the same question, and nothing checked that the two agreed. Two doc comments claimed the runtime called the predicate and that a test policed the agreement; neither was true. Build now returns only the node. The two places that collect ask unifies_its_own_contributions about the child: a window's MergeInput and limitk's pre-selection merge. Every other arm builds its operator and asks nothing, because nothing there collects. A filtering comparison over two scalars selects its carrier side by the rule the runtime's broadcast uses - the side that is not a bound constant - so the two readers describe the same node. Where the choice would be observable they agree anyway: every scalar kind answers Some(true), so such a pair collects nothing whichever side is named. The behaviour test asserts constructed runtime state rather than the predicate: a window over an unknown input or an already-unified child builds no MergeInput, and over a node that reports fragments it does; limitk matches. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry grouping labels across the aggregate split, and decline collapsing rewrites Two wrong answers the label-rewrite admission introduced, both reproduced through query entrypoints before the fix. An aggregate over a label the rewrite creates lost that label: sum by (dst) (label_replace(probe, "dst", "$1", "host", "(.*)")) answered one null-labelled row where the existing plan answers one row per grouping. The partial state carried the scan's tags, so a label the tree creates had no column in it: the producer's schema, the producer's encoder and the consumer's decoder each read a different tag list. All three now use the one list the statement declares, derived once in AggregateFinalize::try_new, so the grouping labels survive the split. A rewrite that can fold two series into one label set went undetected when the two were in different execution partitions, because each partition keeps its own duplicate-position state; an outer aggregate then folded the two rows into a silent sum. The route now admits a rewrite only when its destination is provably fresh for that input - not the metric name, not a tag the input carries, not one an earlier rewrite created - and keeps the existing plan otherwise. That is not a regression: those shapes already used it. A rewrite's collision is observable only where every contribution of a label set runs together, and this route does not yet place rewrite outputs that way. label_join's legal minimum is three arguments (its variadic source list may be empty, which deletes the destination); admission accepted four and left the three-argument form on the existing plan. The regression proves the cross-partition case rather than assuming it: it first shows the two colliding series occupy different execution partitions, then that the collapsing shape is declined and answers what the existing plan answers. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop the canonical-merge pass that no query can reach The insertion pass from the stacked canonical-ownership work never fires in production. Its two insertion points cannot see an admitted tree: a Window only gets one over an evaluated child, and the evaluated-child route is the subquery route this planner refuses before it ever reaches the pass; limitk's single-input form returns earlier, and its multi-input form is refused as depending on input order. So the pass rewrites a tree to insert nothing, and the placement gate that follows is always satisfied by the co-location flag it is given. With the pass go the property model it existed for (NativeProperties, CanonicalOwner, CanonicalGrouping), the single-execution analysis that nothing called, and the placement helpers whose only caller passed a constant. The runtime's own collection is untouched: a window still builds MergeInput for a child that reports fragments, and limitk still merges before it counts. The predicate both of those ask stays, because the runtime asks it - along with `reads_across_series` - and its documentation now describes those readers rather than the pass it used to share the rule with. Restores the parameterized-map validation this work had weakened: a map's parameter count must match its arity, and each parameter must be a scalar on the input's grid. That check is unrelated to canonical ownership and was removed as a side effect. Signed-off-by: discord9 <discord9@163.com> * fix(promql): key the split aggregate's state by the cut aggregate's own labels Restacked on the frame-segment branch after it began deriving native schemas without probing the existing planner, so the shared label walker is now the only source of a tree's output labels and this branch's private copy is gone. The state exported at an aggregate split is keyed by the cut aggregate's own output labels, not the statement's final projection: an outer aggregate that drops its input's labels must not empty the inner groups' keys. Signed-off-by: discord9 <discord9@163.com> * fix(promql): make the without-grouping split actually run, and cover three-argument label_join The `Without` branch of the aggregate split derived its state keys by asking the tree walker for the prefix's labels, and that walker answers nothing for a map that carries a label rewrite. So the split was declined for exactly the shape the comment claimed it handled: sum without (host) (label_replace(probe, "dst", "$1", "host", "(.*)")) answered correctly only because it never reached the native route. The keys now come from the labels the statement declares, which the planner builds from the existing plan and which name a rewritten label; `without` then drops its own names from that set. The split runs, and the regression asserts the physical plan carries `AggregateFinalizeExec` rather than inferring it from a matching answer. That source is only authoritative when the cut aggregate is the statement root: beneath a suffix the statement's labels can be narrower than the cut's, so such a shape still declines and keeps the existing plan. Both comments that described the walker are corrected to say what the code does. `label_join`'s source list is variadic and may be empty, which deletes the destination label. Three arguments are legal PromQL; the route refuses that shape (nothing fresh is written) and must still answer what the existing plan answers, which is now covered. Signed-off-by: discord9 <discord9@163.com> * wip(promql): checkpoint recovered canonical merge wiring Preserve recovered implementation after accidental planner truncation. Not approved for integration: focused integration and planner regressions remain unresolved. Recovery replay and diff whitespace checks pass; functional acceptance remains pending. Signed-off-by: discord9 <discord9@163.com> * wip(promql): checkpoint canonical merge execution fixes Checkpoint at user request; stop further integration runs. Focused integration previously passed, but current full integration remains unresolved. Known blocking issues include nested rewrite fallback and overly broad post-consumer alias handling; this commit is not approved for integration. Signed-off-by: discord9 <discord9@163.com> * fix(promql): restore canonical checks before series consumers Remove post-consumer rewrite alias exemption and unsupported region-gather admission shortcuts. Restore canonical checks for vector folds. Frame plumbing remains a follow-up; no acceptance tests run in this review-first phase. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): simplify canonical merge boundaries Consume canonically routed frames directly instead of a second receiver merge. Remove the new prepared aggregate split and use the existing complete aggregate above assembled frames while preserving parallel scan/rewrite and canonical routing. Keep required evaluated-frame segmentation and identity schema recovery. Signed-off-by: discord9 <discord9@163.com> * fix(promql): complete canonical producer flag simplification Signed-off-by: discord9 <discord9@163.com> * fix(promql): handle count_values labels and vector grid validation Derive count_values output labels from its grouping and generated label so subsequent rewrites can take the native canonical path. Restore VectorOf plan-time grid validation and add focused generated-destination tests; trim unrelated aggregate helper extraction. No acceptance runs yet. Signed-off-by: discord9 <discord9@163.com> * test(promql): quote reserved fixture column Signed-off-by: discord9 <discord9@163.com> * test(promql): escape quoted fixture column Signed-off-by: discord9 <discord9@163.com> * test(promql): assert actual canonical producer routing Distinguish canonical-label routing from ordinary repartitioning through a test-only producer observation hook; remove an incorrect no-router expectation that matched unrelated repartitioning. Signed-off-by: discord9 <discord9@163.com> * test(promql): use physical plan downcast for canonical probe Signed-off-by: discord9 <discord9@163.com> * fix(query): limit frame schema recovery to series frames Signed-off-by: discord9 <discord9@163.com> * wip(promql): preserve recursive segment source binding Signed-off-by: discord9 <discord9@163.com> * fix(promql): confine recursive segmentation to canonical trees Keep ordinary multi-source segment cutting and binding behavior unchanged; only canonical merge trees lower global consumers recursively. Signed-off-by: discord9 <discord9@163.com> * style(promql): order test utility export Signed-off-by: discord9 <discord9@163.com> * style(promql): address focused clippy warnings Signed-off-by: discord9 <discord9@163.com> * style(promql): simplify evaluated grid guard Signed-off-by: discord9 <discord9@163.com> * fix(promql): retain upstream shared-input runtime contracts After restacking, keep duplicate evidence and shared-input propagation intact; only share the canonical collection predicate between windows and limitk. Signed-off-by: discord9 <discord9@163.com> * style(promql): drop unused window binding Signed-off-by: discord9 <discord9@163.com> * docs(promql): align canonical boundary description Signed-off-by: discord9 <discord9@163.com> * test(promql): cover reviewer generated-label collisions Exercise rewrites of labels created by a binary branch and count_values through the native canonical boundary. Restore the base documentation and VectorOf grid validation requested in review. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry partial grouping tags through serialization Bump the derived FusedSeries envelope so a partial aggregate round-trip preserves the grouping labels added by this branch. Signed-off-by: discord9 <discord9@163.com> * test(promql): rebuild serialization fixtures with envelope version 4 Signed-off-by: discord9 <discord9@163.com> * fix(promql): validate aggregate state against grouping tags Signed-off-by: discord9 <discord9@163.com> * refactor(promql): adopt narthex root-field binding cleanup Signed-off-by: discord9 <discord9@163.com> * chore(promql): reset experimental fused plan envelope version Signed-off-by: discord9 <discord9@163.com> * fix(promql): collect label rewrites below sample-dropping maps Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline canonical merges on partitioned tables Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-11-04 19:19:20 +08:00
discord9	b30557960b	feat(promql): preserve scan parallelism through native frame boundaries (#10055) ... * fix(promql): let a closed chain keep its charge until the packer is done Four places released the frame reservation: the early-publish path handed it to the packer, but the queue drain and the end-of-input path freed it in place. Those two run while the packer still holds rows it has not turned into batches, so a series spanning several batches went unbilled once its last piece was handed over. Every emit path now hands the charge over, and the packed stream frees what it holds: what is left when it is polled and pending is the one identity it is turning into batches, and nothing survives its end or its drop. The identity guard is also widened: within one source every series shares its metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is what a range function, a math map and the timestamp shape do -- still maps distinct series to distinct labels. Only rewriting an ordinary label can make two equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame output therefore publish as their chains close instead of buffering a partition. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must Three defects in how this node decided a tree's execution layout. A match whose two operands derive from the same source still needs its match keys co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)` therefore fell through to single-partition execution and lost the parallelism the scan gave it. A raw input never received a matching router: the normalization branch returned it unchanged unless the whole tree was global, while the builder requires a direct matching router for any multi-partition matching-local input. A two-source raw query was rejected at planning. Nothing reconciled the partition counts of different sources either, though the builder requires one common positive count. A single count is now computed for the matching-local case and every router under it uses it, gathering when that count is one. The window restriction also ran before normalization decided the layout, so a tree about to be gathered could be refused for a limit that only applies to partitioned windows. Two tests changed with the behaviour rather than around it: hash routing does not promise which partition holds a given series, so that test compares results instead of placement while keeping its partition count, per-partition execution and operand-order assertions; and a test that pinned the old refusal of a multi-source frame now pins its acceptance. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): keep a raw source's scan partitions instead of gathering them `RawSeriesSourceExec` declared one partition and read every child partition inside `execute(0)`, so a cross-table node lost the scan's parallelism before its inputs reached it. It now declares the child's own count, executes exactly the partition it is asked for, and re-labels that partition's frames. That leaves `SeriesGather` with no consumer: it existed to interleave several partitions into one stream. The module is gone rather than kept as a hidden N-to-one path a later plan could quietly fall back on. Signed-off-by: discord9 <discord9@163.com> * fix(promql): publish a raw source's frames under the ordering it declared `pack_raw_series` derives the frame schema from the stream's own ordering, and `RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing the scan's stream through unchanged broke that agreement: a columnar scan promises key-then-time, so the region published key-then-time frames while the frontend still expected the schema it had built, and `MergeScan` refused them. The forwarding adapter restates the ordering this node declares. It is the weaker promise, so a scan that keeps a finer one still satisfies it, and every other stream property is forwarded unchanged. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw frame that names its labels by its key A region cannot hand out complete labels: it has no metric-name binding, so `prepare_raw_scan` refuses an explicit binding outright. What it hands out is the stored key plus the description a decoder needs, which is the normal input to this node rather than a case to refuse — every single-table selector is planned that way. The node's constructor refused exactly those frames, so any plan that inserted a router under a single-table selector failed before it ran: the matching policies this branch adds, and the one-partition normalization of an evaluated tree. The node publishes labels, because that is what it groups and closes identities by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the input is decoded once, at the unpack boundary this node already has, and an already-labelled stream passes through it untouched. The frame it declares is rebuilt from the input's own declaration instead of being compared against it, which keeps the check that a frame this node cannot re-declare is refused while letting input and output differ in how they name the same series. `frame_schema_with` becomes public for that rebuild, since it is the one place that decides what a declaration is. `FusedSeries` now describes the router's output rather than the scan's when it wraps a raw input, because the binding it hands `build` has to match the frame the router actually publishes. Tested through the node directly: a key-labelled input routes at all; matching puts both pieces of one label set in one owner and closes it once; and `RawPrimaryKey` still hashes the key rather than the decoded labels. Signed-off-by: discord9 <discord9@163.com> * test(promql): measure a routed pair by the width it routes to Four assertions still described the layout this branch replaced: a node with two inputs used to read every partition of both into one and answer one partition, and that gather is gone. The pair is now matched in the partitions it was scanned in, so the fused node answers as many partitions as its inputs route to. Each of these fixtures scans with four partitions, and on the base of this branch all five tests passed, so this is a behaviour change this branch made and did not re-verify rather than a defect: the value assertions behind these shape assertions pass unchanged, in both the native and the legacy comparison. The width is read from the routed children instead of being written as four, and the scan is found by walking down through the nodes that only pass its frames along, since a routed input is no longer a bare `SeriesScanExec`. Signed-off-by: discord9 <discord9@163.com> * fix(query): normalize a multi-partition series-frame root A query engine hands its plan to one stream, and for a plan with several partitions that means `CoalescePartitionsExec` - ordinary batch coalescing. A series frame cannot survive it: one stored series may be split across partitions, and a chunk's `is_last` closes that producer rather than the series, so interleaving two partitions' batches produces a stream whose decoder refuses it (`raw series key changed before final chunk`). Evaluated frames have the same shape of problem: their validator keeps one open chain, and a batch from another partition interrupts it. The engine's root is also a region's root, which is how a cross-table native statement reaches this: its per-region sub-plan is rooted at a partitioned `RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end, before the final sanity check, and it normalizes the root and nothing else - `FusedSeries` already inserts its own owner wherever a consumer needs one. A single-partition root is left alone: one producer is the whole answer. A key-labelled raw root is refused rather than coalesced, because the router checks that one stored key decodes to one label set and it needs the labels to do that. No production plan reaches that rejection today - the only producer of a key-labelled frame is the region's series scan, and every plan that executes it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish explicit labels - and the rejection is what keeps it that way. `schema_check` stays enabled: for the roots this rule accepts, the router re-declares its input's frame with the same content, label mode, decode description and ordering, so the schema it publishes is the one it was given. Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit an evaluated frame as a fused input The extension planner hands a child straight to the fused executor when that child already publishes a series frame, and until now it only recognised a raw one. A segment's output is an evaluated frame, so a statement planned as two segments could not be bound above the first one. The check it replaces asked for a raw frame; the property that matters is that the frame names its own series, which is what `labels() == Explicit` says - evaluated frames always declare it and carry no decode description. Everything else stays with `FusedSeriesExec::build`, which already refuses a mixed-content node, an evaluated input that is not contiguous, and a frame whose declaration does not match the binding it was given. Signed-off-by: discord9 <discord9@163.com> * feat(query): plan a global topk under an aggregate as two segments `sum(topk(1, m))` needs the whole vector for its ranking, which is a property of the tree rather than of the statement, and the routes that existed could only answer a whole-vector tree in one partition: the statement fell back to the existing planner. Splitting it lets the per-series prefix keep the scan's partitions while the ranking sees every candidate. The cut goes below the `topk`, not below the `sum`. A segment holding a `topk` reads across series, so it would gather its own input before evaluating the prefix - correct numbers, no parallelism, which is the thing the split is for. Below the `topk`, the prefix runs on the scan's partitions and the ranking and the fold above it run once, on the child's evaluated frame. Both segments are built from one compiled tree: the child is the prefix subtree, the parent is the original tree with that subtree replaced by an input bound to the child's frame. Nothing is re-parsed, so the grid, the matching rules and the bound scans are the ones the compiler produced. The parent needs its output columns without asking the existing planner to name a computed column after its expression, so `fused_output_tags` now derives the grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels through. It derives output *columns*, not a complete runtime label set: the metric name is bound separately from the tag columns, and a grouping by `__name__` keeps that name in the labels while contributing no column. `fused_root_schema` is that derivation, shared with the ordinary route. The segmenter is its own method, boxed at the call site. Inlined, its future made every frame of the planner's recursion large enough to overflow the stack on a deeply nested statement - `test_parse_or_operator` aborted before this. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the two-segment topk plan and what it keeps The statement's two segments do not land in one physical tree: the prefix has one scan for a child, so the categorizer classifies it commutative and pushes it into the region, while the aggregate above it stays at the frontend. The observer therefore records every plan a query engine wraps, not only the first, so the test can look at the region's own root. What the assertions pin, and why each one is worth its brittleness: - the answer is the global maximum, not a sum of per-partition winners; - the frontend segment holds the complete sum and the global topk over an evaluated input, with no instant evaluation left in it; - the region's segment holds the prefix and its output is an evaluated, explicitly labelled, contiguous frame; - that segment keeps more than one output partition, which is the whole point of cutting below the topk; - the region's root is the series-aware router, one output, and it reads that segment. Column names are not compared: this route names a computed column after its compiled root while the existing planner names it after the expression, which is older than this split and not what this test is about. Signed-off-by: discord9 <discord9@163.com> * feat(promql): share the input-locality classification A tree's distribution requirement is one property, and two layers now ask it. The physical planner already decided with `input_locality` what a tree needs from its inputs; the logical planner has to ask the same question when it splits a statement, or the two can disagree about what a subtree requires. `InputLocality` and `input_locality` are therefore public within the workspace. The classification describes the external inputs a tree needs - not its output distribution, and not which neighbouring operators may share a node - so a caller that reads it as either of those will cut where it should not. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement where a gather would swallow its prefixes The previous commit planned `sum(topk(k, prefix))` by matching that shape. A planner that grows one matcher per query shape grows one per benchmark entry, and the shape is not what decides where a statement divides: the tree already says what each subtree needs from its inputs, and `input_locality` answers it. So a segment is now a maximal subtree with one distribution requirement. Inside a tree that must gather, any child that (a) can cross a boundary - it produces evaluated samples rather than raw samples, aggregate state or a scalar - and (b) does not itself need a gather, becomes a segment of its own and keeps the partitions it was scanned with. Nothing else is cut, and a tree with nothing to cut keeps the plan it had. Every segment rebinds its own inputs: the scans it reads directly first, in the order it first names them, then the segments cut out of it, because `FusedSeries` requires dense source indices over its children. `fused_output_tags` derives the parent's columns from those inputs rather than from the statement's first selector. Three gates keep this from answering what the routes before it would not. Each one is a failure this change produced before it was added: - the existing planner must still be able to plan the statement. Deriving tags locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` - an empty range, which the existing planner rejects - was answered instead of failing; - every segment's tree must construct a `NativeExecutionState`, because nothing below can fall back once physical planning starts, and a grouping by a value field is a shape the runtime refuses; - the tree must contain a prefix that a boundary could carry, so a whole-vector tree with no per-series work inside it is not split for nothing. The scope is one selector. A statement over two tables binds each scan as a `RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in a region, so its prefix would not gain the partitions the cut is for; those statements keep the plan they had, which the acceptance test pins. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the segment boundaries and their scope One statement per shape the cut can meet, asserting what the rule says rather than what one query happens to produce: a per-series tree stays one node, a matching tree routes instead of gathering, a global fold over per-series work cuts that work out, a constant stays embedded because a boundary cannot carry one, and a statement over two tables keeps the plan it had because a segment over `RawSeriesSource` cannot be placed in a region. Answers are compared as sets of rows with the value column renamed. Three things differ between the routes and none of them is the answer: this one names a value column after its compiled root while the existing planner names it after the expression, the two emit columns in a different order, and this one is partitioned so its rows arrive by partition. Comparing them as what they are leaves the values as the only thing an assertion can fail on. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep a selected segment whole A segment is chosen because its subtree needs one distribution; cutting inside it again contradicts the choice. The inner builder was still applying the cut predicate, and that predicate is "exportable and not `SinglePartition`" - which every node inside a per-series subtree satisfies, since a per-series tree has no `SinglePartition` node in it to begin with. `sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one segment, then `abs` was cut out of that segment, leaving a middle segment whose child is a node rather than a scan. That middle segment cannot be placed in a region, so it answered one partition while the scan had several - the prefix lost exactly the parallelism the cut is for. The inner builder now collects the segment's `Input` leaves and nothing else. Signed-off-by: discord9 <discord9@163.com> * refactor(query): bind a segment to its own scan A cut segment carries one source's identity: its scan, its metric name, its time index, and its labels. Building it from the statement's first selector only happens to be the same thing while a statement has one selector. `build_inner_segment` takes the two lists the planner already has - each occurrence's filtered scan, and each occurrence's metric, time index and tags - and binds the segment's single source from them. The parent keeps the wrapped children and the statement-level binding it has now, because a source that reaches the parent directly still has to cross a region as a raw frame. A segment that collects more than one source is declined. `MatchingLocal` satisfies the cut predicate while reading two sources, and one segment cannot carry both sources' identity. No behaviour changes: the entry is still one selector, so no scan is wrapped and the statement-level values are already that selector's. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement that reads more than one table A whole-vector statement over two sources is now cut into one per-series prefix segment per source, with the fold above them in the frontend. Each segment binds its own source's scan, metric, time index and labels, so it can be placed in the region that holds that source, and each prefix keeps the scan partitions it was read with. Three things had to change for the existing segmenter to reach that shape: The cut sat inside `!per_partition && !multi_input`, so a statement over two metric tables - which is `multi_input` - never reached it. The cut now sits under `!per_partition` alone. The aggregate split keeps its own conditions, including `!multi_input`, so a multi-input tree is still not split that way. The single-selector gate was doing two jobs. A statement naming several selectors declines unless those selectors read more than one table: the count alone says nothing about whether the tree has more than one input to place, and the shapes it would otherwise admit - one metric named twice under a fold, such as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers. `cross_table` rather than `multi_input` decides this, because two Mito tables carry no metric logical id and are still two sources. A segment that collects more than one source is declined. A matching subtree over two sources is a cut candidate, and one segment cannot carry both sources' identity, so `sum(topk(1, m / n))` keeps the plan it had. The segmented parent reports the value column the compiled tree names, not the one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where the existing plan reports an expression-shaped name. The values are identical and both names are pinned in the test, so a change to either rule shows up there. Restoring the existing name needs a rename above the parent - the node's own schema must keep the compiled names its output columns are classified by - and is left to a follow-up. Signed-off-by: discord9 <discord9@163.com> * test(query): pin that two sources keep their own scan bounds A segmented statement binds each prefix to its own source's scan, and A1 made that binding: `bare_scans` is derived from the prepared scans after each selector's time filter has been pushed into its own `TableScan`. Nothing tested that the two segments actually read `their` bounds rather than one window applied twice - the existing two-source case has no offset, so both sides read the same range and a crossed binding would answer the same numbers. `sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only in the right side's window, so the two queries must disagree: 20/1 and 30/2 against 20/2 and 30/3. The test asserts they do, which is what fails if both segments were given the left scan's bounds. The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints only the peers: the debug form carries the table each segment reads and the embedded `FusedSeries` segment. An in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence for segments is measured in `native_aggregate_test.rs::check_two_source_segregation` instead. Signed-off-by: discord9 <discord9@163.com> * fix(query): decline a multi-source tree that groups by the metric name The gate that lets a multi-source statement reach the segmenter is `selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`. Two ordinary Mito tables differ in table name and carry no metric logical id, so they are `cross_table && !multi_input`: the guard did not apply and the statement reached the segmenter. `fused_output_tags` keeps a grouping by `__name__` out of the declared columns on purpose - the name travels in the chunk's labels, not as a tag column - so `sum by (__name__)` over two sources derives no tag column while the runtime holds two groups that differ only by metric name. Both are then reported under one label set, as two rows a reader cannot tell apart. That is the failure the metric-logical-table test already documents; it could not catch this path because its tables make `multi_input` true. The check is scoped to a tree that reads more than one source. One source folds one metric, so its groups carry the same name and naming that name or dropping it cannot merge two of them. Signed-off-by: discord9 <discord9@163.com> * test(query): cut a range function on each side of a division The two-source cases so far cut an `Instant` prefix: one sample per grid point. A window function's prefix computes its window inside the region over that source's own samples instead, so the segment carries a `Window` rather than an `Instant` and nothing tested that such a prefix is still a cut candidate. `sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then 4/3. Two different step values are what let a shared window bound change an answer, which is asserted before the result is compared. The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset case: an in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence lives in `native_aggregate_test.rs`. This is the shape of the throttling query in the benchmark corpus: a window under a fold, on each side of a division. Signed-off-by: discord9 <discord9@163.com> * test(query): plan the benchmark's throttling and namespace-ratio shapes Every two-source case so far divides two ungrouped scalars. The corpus queries that motivate this work group by a label on each side, so each segment reports several groups and the parent matches the two sides by that label. Nothing tested that path. `container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total` is the corpus query verbatim: two metric logical tables on one physical table, `sum(increase(...)) by (container)` on each side. It plans as one frontend `FusedSeries` over two `MergeScan` children, each holding a segment bound to its own metric, and reports the same labelled rows as the existing plan. The namespace-CPU-usage-over-requests shape differs from the corpus text only in the metric names, because the corpus names carry recording-rule colons that a table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form and is asserted as that shape rather than as the query verbatim. Both tests assert the native route answered, so a decline fails here instead of passing on the legacy rows. The value column still differs from the existing plan's - the same known gap recorded for the ungrouped case - and both names are pinned. The fixture helper records the three setup traps the hard way: `INSERT` needs explicit column lists because the physical table's column order differs from the logical table's, SQL keywords used as tags need quoting, and no sample may sit at timestamp zero, which both the scan's lower bound and PromQL staleness drop. Signed-off-by: discord9 <discord9@163.com> * test(query): record which native route every corpus query takes The corpus is 29 queries and nothing said which of them the native route answers, or by which of its plans. One parameterised test now runs all of them over generated fixtures and records, per query, the route and the numbers. Four routes, and the frontend tree alone cannot tell them apart: - `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one per source; the prefix work happens in the regions. - `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region `FusedSeries` that reports `partial state:`, so what crosses the boundary is aggregate state. - `region-local` - the frontend holds no native node at all: a per-series tree is commutative, so the whole `FusedSeries` is pushed into the region. A classifier that only looks at the frontend calls these legacy, which is what happened before this test existed. - `legacy` - no native node on either side. All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two `region-local`, none legacy. The two `region-local` ones are the corpus queries that fold nothing - a bare selector and a `rate` - which is also why pushing the whole tree into the region costs nothing. The test asserts only that the existing plan answers each query with rows, and that a native route's labelled rows agree to within `1e-10` relative. It records the route rather than asserting it, because which route a shape takes is a property of the planner that this test exists to report, not to legislate. Ten queries differ from the existing plan in the last bits; the largest relative difference is 2.9e-16, which is summation order, not arithmetic. The table prints that figure per query so a future equality failure can be told apart from rounding without re-measuring. Every fixture here is generated: explicit `INSERT` column lists, quoted SQL keywords, and no sample at timestamp zero. They establish the plans and the values; they say nothing about speed, memory, or how many partitions actually produced rows. Signed-off-by: discord9 <discord9@163.com> * test(query): record how many partitions each corpus query's nodes used The route matrix said which plan answers each query. It did not say whether that plan produced anything, so a node that ran and emitted nothing looked the same as one that worked. `output_values_per_partition` reads `native.root.output_values` off a `FusedSeriesExec` and returns the counts per partition, and the matrix prints them as `partitions used`. `assert_four_native_partitions` keeps its own assertions and now builds on the same helper. What the column does not establish, and says so where it is printed: these fixtures hold a handful of series each, so a scan puts them all in one partition - every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the plan. The one place a fixture is built to fill the scan's partitions is `native_aggregate_test.rs::check_two_source_segregation`, which asserts all four produce values for both segments of a two-source statement. This matrix records a single assertion instead: at least one partition produced values, so a node that ran and emitted nothing fails. Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise divisions. Their region side has no `FusedSeriesExec` at all, because both scans are matched inside one region: that is the route, not a missing measurement, and the table says `no metric` rather than implying zero. Signed-off-by: discord9 <discord9@163.com> * test(query): record the new rule in the explain stages `TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering `SeriesFrameOutput` adds one line to that listing whether or not the rule changes the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary tables, whose schemas carry no series frame declaration, so the rule returns the plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the existing lines say for the same reason. Regenerated by the sqlness runner; the diff is those three lines and nothing else. Signed-off-by: discord9 <discord9@163.com> * test(query): make the corpus matrix prove what it claims Three gaps, all in the test the "every corpus query answers natively" claim rests on. The partition helper returned early on a plan that reported no `native.root.output_values`, so a producer that registered nothing passed on the declared partition count alone. It now expects the metric: the point of the assertion is that every partition executed, and an absent counter says nothing about that. `segmented` was decided by the frontend tree containing a `MergeScan`, which also matches a frontend native node consuming raw frames from its regions - a different placement that says nothing about a prefix running in a region. The merge scan's `Debug` carries the plan its region runs, so the check now reads that for a segment; the raw case gets its own `frontend-raw` route. Four queries move there: L05, L06, L08 and R03. The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been probed in a throwaway run. They are entries now, with the join labels Q08 and Q09 match on and real `le` buckets for Q10's histogram. A fallback is no longer a route to record: a case that reaches the existing plan fails, because that is the regression this table exists to catch. Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29. Signed-off-by: discord9 <discord9@163.com> * refactor(query): let the compiler decide which calls this route answers A second list of function names stood beside the compiler's own capability check, and the two disagreed. `admitted_selector` named five calls; the compiler folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same call meant two different things depending on what wrapped it. The compiler already answers this question: `compile_call` refuses a name it has no kernel for with `NotImplemented`, and the planner turns that into a decline. With the list gone there is one place that decides, and a call the compiler folds reaches the route whether or not it is wrapped in an aggregate. `sort` and `absent` read across a vector rather than one series at a time, which is why the old list named them out. They compile, so they are planned; the tests that asserted the refusal now assert the admission. Only the shapes the compiler genuinely cannot express keep the existing plan - a subquery evaluates its body on a grid of its own, which this route does not offer. Signed-off-by: discord9 <discord9@163.com> * refactor(query): admit an operand by its shape, not by call name An operand of a hybrid binary was admitted only if it was a selector under one of five named calls. The compiler folds many more than those five, so `ceil(m)`, `irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under an aggregate was admitted - the name list disagreed with the compiler about what this route can answer. What the list was really protecting is a layout assumption, and the shape states it directly: `try_plan_native_operand` hands back a selector's planner context, and the hybrid caller reads field columns and tags off that context, so an operand whose output layout is not its selector's cannot be one. `sum(A) / B` is refused for that reason, not because `sum` is a name. `single_selector_spine` states it on the expression: a selector, parentheses, unary, a call carrying exactly one vector-or-matrix argument, or a binary whose other side is a compile-time constant. Calls are followed by the type of their arguments rather than by `constant_value`, because a call's string parameters are constants that `constant_value` does not fold and would otherwise look like data. Which calls this route answers is now the compiler's answer alone: a name it has no kernel for comes back as `NotImplemented`, which the planner turns into a decline. Signed-off-by: discord9 <discord9@163.com> * feat(promql): mix raw and evaluated inputs in one fused tree `build` refused a tree whose children were not all raw or all evaluated, but nothing below that point assumed uniformity: `execute` picks the decoder from each source's own schema, `FusedFeed` carries the source index, and `collect_source_schemas` already validates a repeated source against its own grid. A frame stream has one format per stream, not one per consumer, so one segment can decode the raw samples of a rate and compare the result against a vector a previous stage already evaluated. Remove the refusal and make the remaining per-input decisions per source: `evaluated_global` now asks about the source it orders, and `window_sources` keeps only the evaluated sources, since a raw source is answered by the raw path it always used. Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y` arrives evaluated on the statement grid. The test asserts all ten (host, timestamp) rows and fails on the removed refusal if the refusal is restored. Signed-off-by: discord9 <discord9@163.com> * feat(promql): bind a segment to every source its subtree reads `cut_as_segment` accepts any exportable subtree that is not `SinglePartition`, so a matching subtree over two sources - `m / n` - is a legal cut point. `build_inner_segment` then required exactly one source and returned `None` for it, and `segment_tree` propagates that `None` as "the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back to the legacy path even though its cut was legal. Build the segment from every source the subtree reads. Each child gets the `RawSeriesSource` boundary already used for separate metric tables, which carries that source's own metric, tags and fields across the boundary and makes the child placeable; a segment over two tables cannot sit inside either region, so it runs above them. The single-source case keeps its existing direct-scan path unchanged, which is what lets a one-table segment still be pushed into its region. Field names come from `tree_source_fields` on the rewritten tree, whose source indexes are already dense in the order the children are built. That helper also validates a repeated source against its own schema and refuses a non-dense index set, which a second traversal would have to re-derive. It becomes `pub` and is re-exported for the query crate to call. Take the segment's time column from its own first source rather than from the statement's first: a cut whose sources are `[1, 2]` would otherwise name the time column of a table it does not read. Cover the construction with a planner test and with a real-Mito integration test that asserts both the exact values and the route: one segment over the two tables, each child reading its own table through its own raw series source. Without the route assertion a silent fallback would pass on equal values. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a matching tree at the configured width A matching tree's width came from its inputs: `matching_partitions` took the selected source's count, or the maximum of every input's. In standalone a cross-table statement arrives as one `MergeScanExec` per table with one partition each, so `m / n` matched on a single worker while the scans under it ran in parallel. The engine's `execution.target_partitions` was reachable - `plan_fused_series` already uses it for `series_scan` - but never reached this decision. Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which resolves a matching tree's width as `target_partitions.max(1)` for both the same-source and the multi-input case. `to_execution_plan` keeps its signature and its input-derived width, so a caller that has no configured parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do not consult the value at all: a per-series tree runs on whatever partitions it was scanned with, and asking for a router there would only add a shuffle. Forward the value through `FusedSeriesExec` because `with_new_children` rebuilds from the node's own children, which are the ones below the router; rebuilding without it would resolve the width again from the pre-router count and silently fall back to one. Verified on a real standalone: the two-table segment now reports four partitions with a `SeriesRepartitionExec` under each one-partition region scan, where it reported one before, and the rows are identical. Signed-off-by: discord9 <discord9@163.com> * test(promql): declare a keyed test frame's ordering, and order its keys `plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`, which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it returns packs what `RawScript::ordering` reports, which is now the key-then-time order this helper models. The node reads the ordering off the *plan* schema, so the two disagreeing meant the router scheduled these fixtures as `Contiguous` while unpacking validated them under the stricter ordered contract. Build the schema through `frame_schema_with` with the ordering the stream actually produces, so the plan and the runtime frames state the same contract. That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with both sides claiming `contiguous`, the ordered validation never ran. Running it showed the matching fixture feeding key `b` before `a` within one partition, which its declared ordering forbids. Reorder the partition; the two pieces of `host="a"` still arrive in different partitions, which is what the test is about. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): declare the key order a two-phase raw scan actually keeps `SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a `TwoPhase` scan that is not reading columnar artifacts. The data that path produces is ordered by primary key: - candidates are merged and sorted ascending by the encoded primary-key column, then decoded to ids by walking that ordered stream without reordering; - partition assignment is a deterministic TSID-to-partition map that appends in arrival order, so one series never spans two partitions; - each assignment is drained completely before the next is taken, rather than emitted in completion order; - the raw adapter's predicate buffering preserves input order and cannot let a later key overtake a held one, and two-phase records always report `ends_package: false`, so it never engages the package-tail machinery that legacy relies on. Declaring the weaker order costs two things that are available: the frame codec's `SequenceValidator` never ran its increasing-key check, and a repartition owner could not use a key watermark to release an identity before another input ended. It also contradicts the scanner's own documented contract, which states the output is always ordered by (primary key, time index) within a partition. Report the order the path actually keeps, and reword the two ordering errors that said "legacy" now that both modes can reach them. Cover it with a test that forces two-phase mode and the non-prepared fallback, asserts which of the two it ran, and then checks per-partition key monotonicity, per-series timestamp monotonicity, single-partition ownership, and the ordering the scan reports. The fixture writes three rounds, uses two output partitions and a residual predicate, and keeps one series above a batch so the multi-batch path is exercised. The three existing two-phase raw tests asserted the old declaration. They check that a series stays in one run and never recurs non-adjacently, which the stronger order still guarantees; only their expected declaration and their names said "contiguous", so those change and the substantive assertions stay. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw match only by a key the computation keeps `same_source` answers "this subtree reads one source", and a `MatchingLocal` classification was taking that answer as licence to insert the router below the fused computations, hashing the labels as they are at the raw stage. Those are different facts. A math map drops the metric name and the `__type__`/`__unit__` metadata, `timestamp` does the same, and every range function drops them except `last_over_time` and `first_over_time` — so `on(__name__)` over `abs(...)` routes by a key the very next node deletes. Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side routes `x` and `y` to different owners; both become the empty key after `abs`, nothing re-partitions between the map and the match, and the pair is silently lost. Decide preservation separately from source identity. A node keeps the key when it cannot remove one of the labels the key is built from; a `Window` consults `RangeFunction::drops_metric_name` rather than assuming, so the two functions that keep the name still route early. Anything not proven to keep the key — a label rewrite, a histogram map, an unmodelled node — does not establish preservation, and the tree is classified `SinglePartition`. That is not a fallback that trades parallelism for correctness. When the computation removes the labels the key is built from, every series on one side reduces to the same empty key, so every left series matches every right one: the match is whole-vector by construction and has no partitioned form that preserves its result. One partition is the only placement that answers it. Where the key does survive, the match stays partitioned. Signed-off-by: discord9 <discord9@163.com> * fix(query): describe an admitted operand by what it produces `try_plan_native_operand` handed the binary matcher the context saved during scanning, which describes the operand's *input* layout. An admitted call can change that layout: `histogram_quantile` folds a classic histogram and drops `le`, so the scan context still names a tag column the operand no longer produces, and default matching then builds a join key over a column that is not there. Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)` declines whole-tree planning (two selectors over one table) and is then admitted operand by operand; each operand drops `le`, and planning fails building the join with `FieldNotFound { field: "le" }` against a schema of `timestamp, field_0, host`. Publish only the tag and field columns that the operand's own plan schema actually contains, and drop `use_tsid`, which the produced output does not carry. The check is against the schema the node returns rather than against a list of functions that happen to shrink it, so a call added later cannot reintroduce the gap by not being on a list. Covered by a classic-histogram fixture with `host` and `le` and a float bucket field, asserting each operand's context reports `["host"]` and that the planned join key omits `le`. Reverting the change fails that test with the `FieldNotFound` above. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the partitioned duplicate-group answer, with the gap named `native_hybrid_binary_05_06_08` asserted that the right side's two `job="other"` series are refused as a duplicate match group. On the partitioned route they are not: the check is gated on `lhs.presence && rhs.presence` computed from the partition's own slices, so the partition holding `other` sees no left-hand data and filters every point out. Pin what the route actually answers, and state the three things that constrain the fix, because each has already been tried and rejected here: - the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a Prometheus-generated case where the left is absent at one point and the right carries three samples of one group there, and the answer is a value with no error; - the scan cannot run unconditionally, for the same reason; - the match cannot be collapsed to one partition to make both operands whole, because that gives up the parallelism this route exists for. What remains is to let the verdict see whole-operand presence while the partitions keep answering — a deferred check rather than a different placement. This test now makes any change to the current answer visible instead of silent. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a nested match instead of gathering it A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` — passes the logical admission check, because `independent_scan_pair` recurses through `scan_operand`. The physical locality classification does not go that far: it recognizes a match only when both operands read one source each, so `same_source` fails on an operand that is itself a match and the whole tree is classified `SinglePartition`. The fused computation then ran on one partition while the scans under it stayed parallel, which the existing planner does not do: it builds ordinary equijoins that DataFusion can execute as partitioned hash joins. Admission and placement are different questions, so they now get different predicates. The tree is segmented when it is cross-table, not multi-input, classified `SinglePartition`, and its root is a vector match with a nested vector match directly under it. The inner match becomes a segment and the outer one repartitions the evaluated operands by its own key, which is what the segmenter already does for any exportable evaluated prefix; a nested match whose two levels use different keys is routed twice, once per key, rather than pretending one routing serves both. The predicate stays narrow on purpose. Three-selector trees over distinct metric logical ids are `multi_input` and keep their existing construction, including the parent binding that construction produces. A same-table nested match does not enter this branch either, so its prior admission is unchanged. The plan test's collector walks each fused node's native tree in expression-child order and resolves each input's `source_index` against that node's inputs, so a segmented plan is checked for the same property as before — every selector occurrence bound exactly once, in expression order — rather than being checked more weakly. It panics on a shape it does not model instead of returning an empty list. Signed-off-by: discord9 <discord9@163.com> * fix(promql): prove a raw match key survives the computation `preserves_matching_key` decides whether a subtree keeps the labels a vector match routes by. Two arms answered yes without looking at the field that decides it. A binary with a scalar operand recursed into the vector side alone, but arithmetic and a `bool` comparison drop the metric name while a filtering comparison keeps it — the distinction execution already makes. And an `Instant` recursed unconditionally, while `timestamp(x)` compiles to `InstantValue::Timestamp`, which drops the name too; the check written for the separate `Timestamp` node never covered that shape. Both now read the field that decides: the operator together with `return_bool`, and `InstantValue`. The labels checked are the whole computed set — name, type and unit — because `drop_metric_name` removes all three and default matching does not ignore the last two by itself. Without this, `(x * 2) + on(__name__) (y * 2)` was classified matching-local, the router hashed the raw names before the maps dropped them, and a valid pair could land in different partitions with nothing re-partitioning afterwards. The regression asserts the value is 10 and that the tree is `SinglePartition`, computing the destinations the router would have used (`x0` to partition 0, `y0` to partition 1 of two). With the old arm restored the value assertion fails with `left: [10.0], right: []`. Signed-off-by: discord9 <discord9@163.com> * feat(query): find a nested match through the wrappers around it Segmentation was gated on the root itself being a vector match with a nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never entered: the wrapper or the scalar broadcast sat where the match was expected. Meanwhile the segmenter already walks children and leaves a wrapper in the parent while cutting what it can carry, so no per-wrapper case is needed — the search just has to look through them. Replacing the root check with `carries_prefix` was too wide: almost any multi-source tree contains something a boundary could carry, so ordinary shapes like `some_metric unless on(tag_0) other_metric` were cut for no gain and their inputs stopped being raw sources. So keep requiring a nested vector match, and search for it anywhere in the tree rather than at the root. Wrappers and scalar broadcasts are transparent because the walk visits every child; the nested match itself is still required, which is what keeps the plain multi-source shapes on the plan they had. `cross_table`, `!multi_input` and the `SinglePartition` locality requirement stay, as before: they are what keeps distinct metric logical ids and the same-table shapes out of this branch. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): let a router publish an identity whose chain is complete `SeriesRepartitionExec` published an identity only once every input had contributed it or ended, because an identity may be split across inputs. A per-series region scan assigns each identity to one partition, so on that shape the rule holds every identity until the other partitions reach EOF: a region's whole raw selection is retained before the first series leaves. The node now carries an `IdentityCompletion`, defaulting to the existing merge. `CompleteDisjoint` publishes an identity as soon as its chain closes and keeps the published identities charged in the pool, so a second chain for the same identity fails the execution instead of emitting a partial series as final. The declaration is the producer's, not something a consumer infers from a node's type: publishing early against a producer that does not own its identities would be wrong, so the conservative mode stays the default and every existing constructor keeps it. Signed-off-by: discord9 <discord9@163.com> * fix(query): choose a series-frame root's router from declared provenance `SeriesFrameOutput` normalized every multi-partition frame root through a router that merges identities, which is wrong for a root whose identities are already disjoint: it waits for every scan partition to end before the first series leaves. The rule now selects `CompleteDisjoint` only for a root whose producer declared that property. A raw root qualifies when `RawSeriesSourceExec` was built through `to_per_series_execution_plan` - which also checks the child is a `SeriesScanExec`, so the type alone is not taken as proof - and a direct `SeriesRepartitionExec` root qualifies because it merged the identities itself. Everything else, including a fused frame root, keeps the merging router: a native label transformation can map two source series onto one output label set, so its output identities are not exclusive. The rejection of a key-only raw frame is unchanged and its reason is restated: coalescing cannot produce such a frame, and a router needs explicit labels to check that one key has one binding. Signed-off-by: discord9 <discord9@163.com> * fix(promql): report the labels `absent` names, not the table's columns `fused_output_tags` answered `None` for `absent`, so the caller fell back to the existing planner's column names. Those are the table's tag columns, which is not what `absent` reports: `absent(m{host="x"})` carries the labels its matchers name, and the compiled tree already knows them in `BoundVectorOp::Absent { labels }`. The label set is now read from that field: the named labels are kept in the input's column order, so `absent(native_shapes{host="missing"})` over a table with `host` and `le` reports `host` rather than `ts, val, host, le`. A label the matchers name that is not an input column still answers `None`, because a frame has no column to carry it; `count_values` and `histogram` are unchanged. Caught by the native-versus-legacy value comparison: the extra column made `absent` differ from the existing plan while every value matched. Signed-off-by: discord9 <discord9@163.com> * fix(promql): describe a frame root's emission from what it does `FusedSeriesExec` declared `EmissionType::Final` for a frame output, but `poll_output` hands a frame's evaluated chunk over while its input is still arriving, so the declaration overstated the node to the optimizer. Only a partial aggregate's state is emitted after EOF; every other output keeps its input's emission type. Signed-off-by: discord9 <discord9@163.com> * test(promql): compare the shapes the native route now admits with the existing plan Removing the shapes a name list used to refuse put single-selector `sort` and `sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route. Nothing compared their answers with the existing plan, so the widening was unverified. The new case runs each shape both ways over one metric table. `sort`, `sort_desc` and both `absent` cases compare the whole printed result, which covers column names and values together. `histogram_quantile` compares values row by row and asserts the one known difference instead of hiding it: a segmented statement names its value column from the compiled tree (`val`) where the existing plan names it after the expression that produced it. That case is given a window long enough to produce a row, so it cannot pass by comparing two empty results. The `absent` cases caught a real difference: the native route reported the table's tag columns rather than the labels the matchers name, which is fixed in the preceding commit. Signed-off-by: discord9 <discord9@163.com> * fix(query): publish a fused root's identities early when the tree keeps them The root rule publishes an identity as soon as its chain closes only for a root whose producer declared that each identity lives in one partition, and it knew two such producers: a per-series raw scan, and a router that had already merged its identities. A fused root was always treated as needing a merge, so a region whose segment is a plain instant selection over one per-series scan still held every identity until the other scan partitions reached EOF - `sum(topk(1, m))` is that shape, and its segment is already multi-partition with no label rewrite. `FusedSeriesExec` now answers whether its output identities are disjoint, and the rule asks it. The answer is narrow on purpose: one input, and a tree of nothing but inputs and sample instant selections, over a per-series scan or another fused node that already answered yes. Anything that computes labels is excluded, because a timestamp or a pointwise map can map two source series onto one output identity, and publishing early would then hand over one identity's pieces as if they were different ones. A vector-to-vector binary reads across series and is excluded too. Signed-off-by: discord9 <discord9@163.com> * fix(query): segment a computed matching key's operands instead of gathering them A vector match whose operands compute away one of the matched labels was classified `SinglePartition` and executed on one partition. That is correct for the bare tree - the raw names cannot route a match whose operands drop them - but it also gave up the parallelism the operands still have: `(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so the computed sides can be routed by what they actually carry. The planner now cuts those operands into segments of their own and lets the parent route their evaluated outputs by the unchanged match key. The key itself is not rewritten: `group_key` reads the labels present at that point, so `On(__name__, host)` degrades to the `host` a computed operand still has. Which shapes take this route is decided by one predicate shared with `input_locality`, so the admission rule cannot drift from the locality judgement: a matching site whose two operands are each proven single-source computations, at least one of which does not preserve the key. A failed preservation proof on its own is not enough - it can also mean the proof gave up, as with a parameter that is a compound expression or an operand that is itself a vector match - and the shared doc says so. The same table read twice is admitted for this shape only. The rule that declines several selectors on one table exists to keep folds such as `sum(m) / m` off this route, not because one table cannot be read twice; the compiler already gives each occurrence its own source index, and each cut gets its own input. Distinct partitions are what the two operands need, and they get them. The regression executes both the same-table and the cross-table shape over a four-host fixture and asserts that at least two output partitions produce rows, so the result is not silently produced by one of them. Signed-off-by: discord9 <discord9@163.com> * fix(query): take the early-publication guarantee from the scan's binding The root rule publishes an identity as soon as its chain closes only when the producer can promise that each identity lives in one partition, and the fused answer to that question asked whether its input was a `SeriesScanExec`. The node's type is not that promise: a scan binds a set of logical tags and decodes those into the series' labels, and `try_to_named_series_scan` requires the caller to pass complete tag names but does not check it. A binding that names only some primary-key columns maps two stored series onto one output identity, and early publication would hand one identity's pieces over as different ones. The answer now reads the binding: the label source has to be `PrimaryKey`, and every primary-key column has to be among the bound tags. A source that carries its labels explicitly is not this shape and answers no, as before. Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is false, full coverage is true, an explicit source is false - and by tests for the predicate's two other parts: a tree that computes labels and a tree that is not one instant selection over one input both answer no, and a gated execution shows a completed identity leaving a fused node while the other input is still held. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop the checks the types and decoders already guarantee Two re-checks ran inside the repartition node over facts its inputs had already established, and both cost real work on every query. A raw merge scanned each chunk's timestamps for strict increase, but `RawTimestamps` cannot hold an unordered buffer: its field is private, its only constructor validates, and the accessors are read-only or order-preserving, so no safe caller can build the value this check defends against. The check is gone; the identity, label and shape checks around it stay, because those compare different producers' pieces rather than one chunk's own timestamps. An owner rebuilt each chunk's identity to prove a chain does not change it, but the decoder that hands the chain over already refuses that: `SequenceValidator` holds one open key and rejects a key or metric change before the chain closes, `EvaluatedSequenceValidator` does the same for labels, and the input task that assembles chains only reads from those decoders. What stays is the identity the owner needs anyway to pick a destination, now built by the named functions rather than inline. Where a schema was needed, the node built an empty stream of its own, wrapped it in a packer and read the packer's schema back. The frame module already answers that question from the declaration, so the fake stream is deleted along with its half of the roundtrip. Signed-off-by: discord9 <discord9@163.com> * fix(promql): recursively segment native matching plans Signed-off-by: discord9 <discord9@163.com> * fix(promql): derive native schemas without legacy probes Signed-off-by: discord9 <discord9@163.com> * fix(promql): recognize complete metric scan identities Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23) * fix(promql): release completed disjoint identity history Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c) * test(promql): assert native aggregate schemas independently Signed-off-by: discord9 <discord9@163.com> * fix(promql): identify internal metric keys by reserved id Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep parallel sources and derive group match labels Signed-off-by: discord9 <discord9@163.com> * test(promql): observe the frontend fused root in scan-parallelism checks Signed-off-by: discord9 <discord9@163.com> * fix(promql): release duplicate evidence immediately on abort Signed-off-by: discord9 <discord9@163.com> * test(promql): preserve operand identity across matching repartition Signed-off-by: discord9 <discord9@163.com> * fix(promql): charge the final duplicate verdict until it is reported Signed-off-by: discord9 <discord9@163.com> * test(promql): observe native scans across frontend and region plans Signed-off-by: discord9 <discord9@163.com> * chore(promql): drop dead test helper and clippy lints Signed-off-by: discord9 <discord9@163.com> * fix(promql): select limitk by canonical label order Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep data-free subtrees in the parent segment Signed-off-by: discord9 <discord9@163.com> * test(promql): prove the duplicate error crosses matching partitions Signed-off-by: discord9 <discord9@163.com> * test(promql): drop debug prints from the segmentation test Signed-off-by: discord9 <discord9@163.com> * revert(promql): keep limitk's Prometheus input-order semantics Signed-off-by: discord9 <discord9@163.com> * chore(promql): satisfy clippy in fused root schema Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-11-04 14:12:56 +08:00
discord9	0d75655ed6	feat(promql): bound metric discovery by the caller's table grants (#10107) ... Discovery enumerates the schema's logical metric tables, which is a wider read than a caller's grants necessarily allow. The frontend now answers "which of these may this request see" before the planner touches the catalog, and the planner filters on that answer rather than on its own enumeration. The scope is process-local state on the query context, skipped by the builder the same way the other local-only fields are: a remote node must not receive a scope that widens what it may read. It defaults to `Unavailable`, so a path that did not prepare authorization cannot discover, and the validator turns "should have been set and was not" into an internal error instead of a silent unfiltered read. The named tables of a mixed query are checked too. The complete extractor answers "no tables at all" when any selector is unresolvable, and that was exactly the selector that made the query enter the discovery branch - so `beta_total{job="api"} or {job="api"}` used to lose `beta_total` from the check. Both the allowed and the unrestricted arms now take the partial answer. A denied metric is skipped rather than failing the query, which is what makes discovery mean "every metric you can see". The split route rejects a query whose name matchers reach a denied table, so the same query answers differently depending on the request header; that difference is stated where the skip is made. The planner's retain runs before any source is prepared, so a denied table's name reaches neither the plan nor an error message. It filters with borrowed keys, one pass to build the set and one lookup per metric. Signed-off-by: discord9 <discord9@163.com>	2026-11-04 12:14:51 +08:00
discord9	74deaf2e90	feat(promql): scan every metric a nameless selector matches (#10106) ... feat(promql): answer a selector that names no metric by scanning each it matches A PromQL selector with no metric name used to be a planning error. It is now answered by enumerating the schema's logical metric tables, compiling one source per match under one union input, so the fused producer still reads what the compiler prepared: one compiler source per selector, not per metric. Each source names the metric it reads, in its own tag list. That is what makes this work at all: two metrics whose ordinary labels are equal are told apart by name alone, and nothing else in the response can carry that identity. Where the name survives is then a property of the operator rather than of discovery, so the same tag walk decides it for every query - the walk now drops the name where the runtime drops it, for a range function, a math or histogram kernel, a parameterized map, a timestamp selection and an arithmetic or `bool` binary operation, and keeps it for a filtering comparison. `sum by (__name__)` keeps it by grouping. The walk already had that same divergence without discovery: no metric-engine logical table carries `__name__` as a column, so a named selector's tags did not include it and the walk's answer was incomplete in the same way. Discovery only made the column real. The metric engine checks that two logical metrics agree on their time column's unit, not on its name, so each source's bound is built from the column that source actually carries, and a source without one is refused rather than read unbounded. Two upstream contracts assumed one table per selector and are widened rather than bypassed: the time-bound pushdown walks a union's sources, and a fused node may declare `__name__` when its inputs each name their own metric instead of binding one name itself. Declining still means the query goes to the existing planner; shapes this route cannot derive keep the plan they had. Signed-off-by: discord9 <discord9@163.com>	2026-11-04 12:11:49 +08:00
discord9	327d754984	fix(promql): let native compilation and execution decide admission (#10071) ... Replace legacy output probing and selector allowlisting with native capability checks. Signed-off-by: discord9 <discord9@163.com>	2026-11-03 20:02:07 +08:00
discord9	3d680a38e9	Merge branch 'narthex' into feat/narthex-serde-native-tree ... #10081 changed one test call site in native/wire.rs. This branch deletes that file, so the resolution keeps it deleted: the codec it implements is what this change replaces. Signed-off-by: discord9 <discord9@163.com>	2026-11-03 17:48:41 +08:00
discord9	577508b66b	feat(promql): decide a duplicate match group across partitions (#10081) ... * refactor(promql): report every duplicate point from overlap `overlap` returned only the first point where two members of a group are present and a caller-supplied predicate held, so the caller could not tell which other points had a duplicate. Turn the predicate into a callback and invoke it for every such point in ascending order; the caller decides what a duplicate point means. The one caller reproduces its previous behaviour exactly: it ignores points its gate excludes and keeps the smallest point per group, which is what the group fold already took. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): number the binary runtimes of a tree `RuntimeNode::build` walks the compiled tree in pre-order, so a counter threaded alongside the input ids numbers each binary node deterministically: the same tree yields the same index on every partition, because every partition builds from the same compiled tree. That gives per-node execution state a key without depending on pointer identity. Signed-off-by: discord9 <discord9@163.com> * feat(promql): reduce duplicate findings across partitions A partitioned match cannot tell whether the other operand has a sample at a point, because each partition holds only its own slice. This adds the reducer that makes that decision possible without deciding anything yet: per binary node and field, it merges the operands' presence as partitions publish it and keeps the duplicate findings they report. Findings are retained regardless of what the gate says at the moment they arrive. A partition that publishes its presence late must not cause an earlier partition's finding to be discarded, so the gate is applied only when the merged presence is asked for a decision, with the same formula the kernel uses locally: `lhs && rhs`, or `lhs || rhs` where a fill value makes a point live with only one side. Presence and the retained findings are charged to the pool as they are added, by their real owned bytes, and a refusal is reported rather than swallowed. Storage is bounded by the grid length: one presence flag per point per operand and at most one finding per point, replaced only by one with a lower rank. Nothing constructs this yet. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): carry duplicate evidence down to the kernel Plumbing only: `FusedSeriesExec` holds a plan-instance slot for the execution's duplicate evidence, `NativeExecutionState` and every `BinaryRuntime` carry a handle to it, and the three vector-match layers that reach `binary_field` accept it. Nothing reads or writes the evidence yet — every call site passes `None` or forwards what it was given — so the crate behaves exactly as before. The slot starts empty in `build` rather than in `execute`: the partitions of one execution are separate `execute` calls on the same plan instance, so a slot created per call would give each partition its own state, while a rebuilt plan is a different execution and gets a fresh empty slot from its own `build`. `BinaryRuntime` gains a `slot(field)` accessor for the index the compiler already assigns it; the next change is its first caller. Signed-off-by: discord9 <discord9@163.com> * feat(promql): publish what a field's match saw, without deciding yet `binary_field` now publishes each side's per-point presence into the execution's evidence and records every duplicate point `overlap` reports, including the ones the local gate excludes. Recording them is the point: a partition whose finding arrives before the other side's presence is known must still be able to fire once that presence does arrive, so the gate is applied when a decision is asked for, not when the finding is made. Each published field carries its node and field index, which is what keeps two binary nodes or two fields from sharing presence and findings. The message a recorded candidate carries comes from the same helper the immediate error uses, so a deferred report and a direct one say the same thing. Nothing decides anything yet: `evaluated` is still computed from the local presence vectors exactly as before, and `settled` has no caller outside the tests. `FieldEvidence` groups node, field and handle so the kernel signatures gained one parameter rather than three. Signed-off-by: discord9 <discord9@163.com> * feat(promql): report a duplicate match group found on another partition A partitioned match decides whether to check a duplicate group from the presence of the other operand, and a partition only holds its own slice of that operand. So the partition holding `job="other"` had no left-hand data, skipped every point, and answered where Prometheus refuses the query. The findings a partition makes are now decided once the whole execution has reported. Each partition publishes its operand presence and records every duplicate point it sees, including points its own gate excludes; the partition that finishes last holds complete evidence and asks for the earliest point that is eligible once all the presence is merged. Earlier partitions return their rows and their stream ends normally, so nothing waits on a partition that is never polled — an abandoned execution simply never reaches a verdict, and a failing one aborts its claim rather than being counted as having contributed. Presence is merged per binary node and field, so two nodes or two fields cannot share a verdict, and `filling` is recorded once per slot: a second publisher that disagrees is an error rather than a silent pick. The gate itself is unchanged, which is why a point whose other operand is genuinely absent still answers instead of erroring: the merged presence still says that point is not live. Signed-off-by: discord9 <discord9@163.com> * fix(promql): size duplicate evidence by the grid, not by partition count `DuplicateEvidence::new` stored the output partition count as its grid length. Those are unrelated numbers, and the caller passed `partition_count`: `publish` kept the operand presence for at most that many points and `record_candidate` returned success for anything past it. With two partitions and a five-point grid, a duplicate group at point four was discarded by the very check that exists to find it, and the query answered. Take the point count from the tree's evaluation grid and the partition count separately. Every claim now leaves the counter exactly once, whether it completes or aborts, and the exit that closes the last claim releases what the evidence holds: the verdict is kept only when the run was not aborted, then the slots are cleared and the reservation released. Previously the last completion shrank the reservation while the vectors and strings stayed allocated, and an aborted claim could leave another claim's exit unaccounted, so the charge outlived the run. Candidate storage becomes a sparse map keyed by point instead of a grid-length vector, charged before it grows: the group's strings, its vector capacity, the candidate itself and a per-entry estimate. A grid thousand points long with one duplicate no longer pays for a thousand empty slots, and the charge now covers the containers it claims to cover. The execution test evaluates a five-point range at parallelism two — more points than partitions — with the duplicate at the last point, which is exactly the boundary the partition-sized array dropped. Its control runs the same duplicate against a left side that is absent at every point and still answers. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decide a partitioned duplicate from the whole execution's presence A partitioned match decided whether to check a duplicate match group from the presence of the other operand, and a partition only ever holds its own slice of that operand. `left{job="api"} / on(job) right` with two `job="other"` series on the right returned `5.0`, because the partition holding `other` had no left-hand data and its gate was false at every point, while Prometheus checks every group of the one side whenever the other side is non-empty at that point. Each binary evaluation now publishes both operands' presence and records every duplicate point it sees, including the points its own gate excludes, because a finding that arrives before the other side's presence is known has to be able to fire once that presence does arrive. The partition that finishes last reduces the evidence and asks for the earliest eligible point. Presence and findings are keyed by binary node and field: a tree can hold several binary nodes, and the kernel evaluates fields separately with presence read from that field's own operand. No partition waits on another. A slot takes its length from the operands it is published with rather than from the statement's grid, because a binary inside a subquery runs on that subquery's own grid, which can be longer. A partition that publishes a different length for the same slot is an error instead of a truncation. The evidence exists only when the node has more than one output partition and the tree holds a binary with two vector operands; otherwise the local gate already sees every operand and there is no verdict to reach. A claim for a partition this generation already issued, or one arriving after it finished, starts a new generation, so a second execution of the same plan does not join an accumulator whose claims were spent. A retained finding holds its match group behind an `Arc` with the side and timestamp, and the error text is built only when a verdict is reported, so recording a duplicate point costs no formatting. Signed-off-by: discord9 <discord9@163.com> * perf(promql): record a duplicate only where this partition cannot decide it A point whose merged-in-partition presence makes it evaluated already aborts the execution from the partition that sees it, so storing it in the shared evidence took the lock to record a verdict nobody would ask for. Only the points whose local gate is false are recorded now, which is exactly the set whose eligibility depends on presence this partition does not hold. Both readings are in the comment at the call site, because the previous round's test depended on recording every point and the next reader should see why that changed. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decide an evidence generation from its state, not from call order `claim_evidence` inferred an execution boundary from "this partition has not claimed yet". Cancellation breaks that inference: an execution that claimed partition 0 and was dropped leaves an unfinished generation behind, and the next execution claiming partition 1 joins it; only when that same execution reaches partition 0 does a new generation start. The two partitions of one execution then sit in different accumulators, and a duplicate that only the merged presence can decide is missed. No two executions have to be alive at once for this. A generation is now dead when it has finished or when any of its claims was aborted, and a claim for a partition that a live generation already issued is an error that names the cause instead of silently starting a new one. The previous review allowed refusing re-execution outright, and this is that: any inference from call order is a guess that cancellation can invalidate. The consequence is stated where it is caused: a plan instance whose first execution claimed some partitions and never claimed the rest has an unfinished generation that no later claim can join. That is the price of refusing re-execution; a legitimate second execution would need an explicit execution lifetime rather than a heuristic over partition indices. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep duplicate evidence single execution Signed-off-by: discord9 <discord9@163.com> * test(promql): pin disjoint duplicate groups and trim unused ranking Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-11-03 15:40:51 +08:00
discord9	ce3ef72faf	test(promql): exercise depth fallback with a legacy-plannable query ... Signed-off-by: discord9 <discord9@163.com>	2026-11-03 12:34:44 +08:00
discord9	851f59428d	perf(query): reuse root plan encodability result ... Signed-off-by: discord9 <discord9@163.com>	2026-11-02 18:26:42 +08:00
discord9	8348d3d2d4	fix(promql): let a too-deep tree fall back instead of failing the query ... The depth limit was reported as `DataFusionError::Internal`, the error a compiler defect gets. The planner only keeps the legacy plan on `NotImplemented`, so enabling the native route turned an answer into an error: a statement compiling to more than 64 nodes -- 70 `time()` terms compile to 139 -- failed outright, where the legacy planner answers it. Switching a feature flag on must never make a query stop working. The two rejections now say what they mean. Exceeding the depth limit is "this statement is beyond what this route handles", which is `NotImplemented` and makes the planner fall back, from both the constructor and the deserializer. Every other rejection in `validate()` -- a mismatched binding, a map over a scalar, a window with the wrong grid -- stays `Internal`, because those mean the compiler built something inconsistent and reclassifying them would hide real defects as unsupported queries. A fallback test pins it: 70 `time()` terms must plan exactly as the legacy route does, with the node count in the message so a future change to the cap explains itself. `postcard` moves into the external-dependencies block. It is declared with `default-features = false`, and that block exists precisely for inherited dependencies whose features are narrowed. Signed-off-by: discord9 <discord9@163.com>	2026-11-02 13:34:08 +08:00
discord9	52260c762f	fix(promql): stop carrying an Arrow schema in the fused payload ... The payload carried a full `arrow::Schema`, which dragged in `arrow_schema::DataType` -- whose `List`, `LargeList`, `FixedSizeList`, `Struct` and map variants recurse without bound through a derived `Deserialize` that cannot be hooked from outside. The tree has a measured, symmetric depth limit; the schema had none on either side, so a small payload nesting `List` a few thousand deep would exhaust the stack during decode, before any post-decode validation could run. Review reproduced that with an 80 KB payload and 8000 levels. The schema did not need to travel. `fused_output_schema` builds every column from three flat inputs and hardcodes every type -- the time column is `Timestamp(Millisecond, None)` non-nullable, values are `Float64` nullable, tags are `Utf8` nullable -- and a frame node's schema comes from `series_frame_output_schema` over the tree's root schema. The only genuinely irreducible part is which columns are labels. So `output_tags: Vec<String>` replaces `output_schema: Schema`, and decode reconstructs the schema instead of trusting one. That is strictly stronger than validating carried data: a mismatched schema can no longer be expressed at all, where before it could be carried and merely checked. The three formats differ, and the difference is now meaningful rather than decorative: - a frame node carries no tags, because its label columns come from the tree; decode rejects a non-empty field instead of comparing a copied value to itself, which is what the previous revision did; - a partial fold carries `binding.tags` and decode requires the exact order, because the state layout is position-sensitive and the binding is the ordering authority; - a plain-rows node carries its output tags, the one case where the labels cannot be derived from the tree. `skip_encodability_gate` in the district analyzer was one flag doing two unrelated jobs: skipping the Substrait encodability check, and bypassing multi-input column requirements on the strength of a placement proof. Making ordinary and partial producers serializable removed the premise of the first while leaving the second intact, so they are two fields now -- `skip_in_process_encodability` and `in_process_fused_root` -- and the placement bypass reads as `has_fused_pair_placement_proof` rather than as "we skipped the serialization check". Signed-off-by: discord9 <discord9@163.com>	2026-11-02 12:40:20 +08:00
discord9	e721794d37	fix(promql): bound tree depth on both sides and validate merge inputs ... The depth limit was decode-only and set to a number nobody had measured. A 129-deep tree could therefore be built, serialized and shipped, while the receiver refused it; and in CI the boundary test itself aborted with a stack overflow, because a 2 MiB thread was asked to build, serialize, decode and drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is not covered by a guard that only wraps `Deserialize`. Measured instead of guessed. A probe walked build, serialize, decode, validate and drop at increasing depth inside a named 2 MiB thread: 32 through 128 completed, 160 aborted. The cap is now 64, half the highest passing depth, and the same number is enforced in `NativeTree::try_new` and in the deserializer. Verified through the real constructor: a 64-node tree round-trips, and a 65-node tree is refused when it is built as well as when it is decoded, so sender and receiver accept the same set. The boundary test runs on a named thread, which is how a future failure will identify itself. The merge node accepted any single input. `with_exprs_and_inputs` rebuilt `AggregateFinalize` over whatever it was handed, on the reasoning that a rewritten input only changes where the state comes from. Nothing checked that it was state at all -- and until now nothing could produce a counterexample, because a values-producing producer could not be serialized. The serializer was standing in for a validation that belonged to the node. `with_exprs_and_inputs` now rejects an input whose schema is not `state_schema(binding.tags)`, so substituting a computed-values producer fails at the logical node with a message naming the expected schema, rather than depending on an encoding limitation. `AggregateFinalizeExec` already had the same check at physical planning; the serializer stays permissive. The three placement tests in `commutativity.rs` asserted the old refusal as their mechanism -- an ordinary fused node "fails to encode", a partial producer "cannot be pushed down". Those expectations were the encoding limitation wearing a test's clothes. They now assert the responsibilities that remain: an ordinary fused node round-trips through the encoder, a partial producer rides to the region while its finalize stays on the frontend, and a values producer handed to a merge is refused by the schema contract. The test-only `NPRQ` envelope is gone. It framed trees with its own version, a 1 MiB cap and its own rejections, none of which production had, so `native_tree_envelope_rejects_version_size_trailing_and_truncation` was proving a helper's protocol while the PR description claimed a production guarantee. Trees are tested through plain postcard, transport through the real `FusedSeries` interface. The redundant clone-and-revalidate in `FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already routes through `try_new`. Signed-off-by: discord9 <discord9@163.com>	2026-11-01 16:40:35 +08:00
discord9	72bee4ae9c	feat(promql): answer whole-vector and multi-metric statements natively (#10025) ... * feat(promql): name a fused tree's tags from the existing plan A fused node declares exactly the tags the planner gives it, and reads each one back off the series, so a label set this route cannot derive is a label set the answer would lose. The walk over the tree derives labels for a match, a set operation and the point-wise wrappers over them, and names none for an aggregate's groups or a whole-vector operation - which today declines every such statement. Take those columns from the existing planner instead, the way the aggregate split already does: its tag columns are what it projects for the same statement, and they are now shared through one helper rather than written out twice. Only the tag columns are copied, because the node names its field columns after its own tree while the existing planner can name a computed column after its expression. A label rewrite is a shape the walk cannot derive at all: label_replace adds its destination label and an empty replacement removes it, so passing the input's labels through is an inference the rewrite contradicts. Report no label set there. The rewrite itself keeps the existing plan, and this does not change that: the native runtime refuses two series a rewrite gives one label set, where that plan answers them, so admitting the shape would turn an answer into an error. The test that holds this now covers the two-occurrence shape as well. An aggregate or whole-vector root does not reach this branch today: both read across series, so they are planned by the aggregate split or fall back. It is the prerequisite for the placement that runs such a tree in one execution, and reaches the branch when that lands. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a classic histogram function above the aggregate cut The split accepts a suffix that runs once over the fully merged series, which is why TopK/BottomK were already admitted: the merge has folded every partition's contribution to each group, so the series the suffix reads are the ones the statement's input would have produced in one execution. A classic histogram function is that kind of operation. It reads each histogram off the bucket series that share its other labels and drops le, and the merge has already folded each (..., le) group into one complete series on the evaluated grid, so the buckets it reads are exactly the ones the input would have had. Reading le off those merged series is the same computation, not an approximation. LimitK still declines: it depends on the storage input order, which the merge does not preserve. This is what the o11ybench query `histogram_quantile(0.99, sum(rate(...[5m])) by (instance, le))` needs: it was planned by the existing path only because the split refused a whole-vector operation above the cut. The acceptance test runs the shape over real data and compares the rows two executions report: the fixture spreads six bucket series over several files and a memtable range, so the partial sums genuinely have partitions to merge, and the native answer must equal the complete input's for quantiles at both ends of the interpolation and for histogram_fraction. It also asserts the native run really merged state, so a silent fall back cannot pass it. Signed-off-by: discord9 <discord9@163.com> * feat(promql): read two metric logical tables as two inputs A two-selector statement whose two sides are aggregates kept the existing plan, even where both sides read a metric-engine logical table. The o11ybench query `sum(max by (...) (a)) by (namespace) / sum(b) by (namespace)` is that shape, and it fell back for three separate reasons. First, the engine gate refused every multi-selector metric-engine statement. The clause after it already requires each such selector to be a recognized logical rewrite, so the removed clause refused only the case this change handles. Second, a metric logical table is rewritten to a scan of the physical table restricted by `__table_id = <id>`, so two logical tables over one physical table prepare two scans of the same table. The check deciding whether the node binds one metric or one input per occurrence compared table names only, so the two metrics read as one and both inputs would carry the first one's name - a wrong answer, not a fall back. The id the rewrite already requires is now part of that identity. Third, a tree over two aggregates reads across series, so it is not answered per partition, and the branch for such trees was the aggregate split, which takes one child. A node with more than one input needs no split: it declares one partition and reads all of every input's partitions in one execution, so every group is folded in one place. Such a tree now falls through to the node. The one-selector path is unchanged: it still requires `can_split` and still declines a tree the split cannot cut. So is the multi-selector case whose inputs stay direct scans of one table - such a node is not region-local, as `Categorizer::check_extension_plan` refuses a cross-series plan over more than one scan - which is why the condition is the metric-logical-table one. The acceptance test runs both shapes over two logical tables of one physical table, with values that make a swapped input visibly wrong, and compares the rows the two routes report. Signed-off-by: discord9 <discord9@163.com> * test(promql): log the metric pair's rows through telemetry, not stdout The acceptance test printed its evidence with println!, which the crate lints as an error under -D warnings, and echoed a Display value through to_string inside the format arguments. Every other test in the file logs through common_telemetry::info!, so this one does too. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a tree the runtime cannot execute The multi-input route skipped the aggregate split, and the split was what refused a tree the runtime cannot execute: `check_cut` declines a `quantile` and a grouping by a value field. Skipping it turned queries the existing plan answers into errors, because the node is chosen while planning the logical plan and a refusal the physical layer raises later has nowhere to fall back to. `quantile(0.5, a) / sum(b)` over two metric logical tables reported `NotImplemented: the native runtime does not execute the Quantile { .. } aggregate yet` where the existing plan answers it, and `count(count by (val) (a)) / sum(b)` failed on the grouping by a value field, which the outer `count` had hidden from the column check. Ask the runtime before taking the route, with the same construction the physical layer validates by, and keep the existing plan for whatever it refuses. Asking it rather than listing the unsupported kinds is what makes the check cover refusals this planner does not know about. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a grouping and a selection the node cannot answer A multi-input node skips the aggregate split, so the refusals the split makes for reasons that are not about what the runtime computes were skipped with it. Two of them matter here. A grouping by the metric name keeps a label no column of the existing plan can name. The native aggregate folds its groups by `__name__` as by any other label, but the label set the node declares comes from the existing planner's columns, and `__name__` is the name a raw frame carries rather than a column of any scan - so the existing planner's `by(__name__)` finds nothing and reports no label. The schema would then report one label set for two groups the tree keeps apart: `sum by (__name__) (a or b)` over two metrics would answer two rows a reader cannot tell apart, which is a series identity lost rather than an answer given. A selection that depends on the order its input arrives in is the other. `limitk` keeps the first `k` series of each group in input order, while a node with more than one input reads its inputs' partitions through a gather that hands them over in the order they became ready: the schedule would decide which series a query keeps. Prometheus defines the selection over a series set. The split declines `LimitK` for the same reason. Every other vector operation orders its input itself - `topk` by label, `sort` by value, `limit_ratio` by a label hash - which is why only this one is asked about. Both shapes keep the existing plan. `limitk` is not answerable by either route today, since the existing planner has no kernel for the operator; the point is that the native route is not the first to answer a selection the schedule would decide. Each guard is checked with the guard removed, so both tests are known to fail without it. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline unsupported count_values groupings before execution Signed-off-by: discord9 <discord9@163.com> * docs(promql): scope the native capability probe guarantee Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the same-table pair the multi-selector exclusion used to keep out State that a Mito input joins a multi-input tree only when a second metric logical id makes the tree multi-input, and that only LimitK's selection depends on arrival order. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline ranking grouped by a value field before native execution topk, bottomk and limitk key their groups by series labels, while the existing plan also accepts a value field there and forms one group per sample value. Admitted multi-input trees could therefore keep one row where the existing plan keeps one per group. Validate the grouping when the vector runtime is built so the statement keeps the existing plan. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-10-29 20:16:44 +08:00
dennis zhuang	9f906da091	feat(promql): run time(), vector(s) and data-free scalar broadcasts natively (#9988) ... * feat(promql): run time(), scalar(v), vector(s) and dynamic scalar broadcasts natively Bind time() and zero-argument date functions as EvalTime, scalar(v) as ScalarOf and vector(s) as VectorOf, and run binary operations whose scalar side varies per grid point: the scalar is read whole, vector chunks that arrive before it are held and charged, and both operand orders and bool comparisons follow Prometheus. A tree holding scalar(v) runs as one execution over every scan partition, and the planner admits it only over an unpartitioned Mito table. Data-free statements plan without a scan. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): hold the labels of chunks waiting for a varying scalar Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(promql): read round and clamp parameters from varying scalars natively Bind round, clamp, clamp_min and clamp_max whose parameters are not all constants as a ParameterizedMap with scalar parameter nodes, run through map_math_with_parameters. The parameters are read whole first; vector chunks that arrive before them are held and charged, through the same hold the varying-scalar broadcast now shares. Constant parameters keep the MathOp map. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin a parameterized map's empty closing chunk Also correct the held-chunk docs, use same-table examples for varying map parameters, and assert the native decline instead of the existing planner's failure. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): plan vector(s) natively whatever the table's field is named The fused route declined a root field that is not a raw field, which is every root over vector(s): it names its result `value`. Take the compiled root's fields as they are and decline only a name the output already gives a tag or the time index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): reserve a constant map parameter before building it Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): leave scalar(v) to the existing planner until it has its own execution boundary Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): leave data-free statements that generate labels to the existing plan Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): reserve the output point-wise kernels build before running them Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-28 20:07:45 +08:00
dennis zhuang	5b17bbd584	feat(promql): run info() natively (#10005) ... * feat(promql): run info() natively info(v, {selector}) fell back to the existing plan, which has no info and refuses the query ("Unsupported expr type: info", or "Cannot accept multiple vector as function input" when a selector is given). Compile it into the native tree and run it on the info kernel: - Plan: NativeExpr::Info { base, info, selector }. The info side is an instant selection reporting each selected sample's own time in milliseconds (InstantValue::InfoTimestamp, which replaces the BoundInstant.timestamp flag); validation refuses any other info side. - Compiler: info_input derives the info selector the way Prometheus does: target_info or one __name__="..." matcher, the data matchers, and the @ and offset every selector of the first argument shares (subquery offsets added until an @ anchors them, none when they differ). Regex or negative name matchers, which can select several tables, __field__/__schema__/__database__ matchers and multi-field info tables are refused. The planner's selector walk uses the same helper, so both bind the info table as the last input. - Runtime: BinaryRuntime runs the kernel as PairKernel::Info, with the retention and pool accounting of binary and set nodes; the kernel reports its workspace through info_with_reservation. - Planner: the info table is a second table, so it is read through its own RawSeriesSource and the node runs on the frontend. The output adds the info table's tags, or only the data labels the selector names. Partitioned or non-Mito tables and a missing info table keep the existing plan. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin region-local placement of info over its own table info(rate(target_info[5m])) reads the info table twice. Check that the planner binds both as direct scans under one FusedSeries node bound to the table, that the node is placed in the region, and the columns it reports. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep sample-less base series out of info's identifying patterns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): share identifying label strings in info() instead of copying them Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-28 10:39:43 +08:00
dennis zhuang	654a44aa9f	feat(promql): run native statements over any number of selectors (#10004) ... * feat(promql): run native trees with more than two selectors Lift the two-selector cap of the native PromQL route. A tree of one-to-one vector matches and set operators, nested under point-wise wrappers, is now admitted for any number of selectors: on one table the node runs in the region over every scan, across tables it runs on the frontend over one RawSeriesSource per selector. Partitioned tables, aggregates, whole-vector operations and many-to-one matches still keep the existing plan. The dist analyzer and commutativity checks accept a same-table node with more than two scans; without that such a plan fails in the analyzer. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover nested set operators and metric-engine multi-selector decline Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): wait for aborted partition tasks to release their inputs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit 3f8699782c3116c4063ec70a3522d1f5f3bff897) * docs(promql): say spawned partitions are released after an error, not before Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 09:18:26 -07:00
Dennis Zhuang	02eb3aa60c	chore: merge narthex into feat/narthex-cross-table ... Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 18:20:21 +08:00
discord9	b29a471575	fix(promql): repair native execution contracts and remove obsolete paths (#9980) ... * fix(promql): restore native scan statistics and remove unused raw merger Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): repair native execution contracts and remove obsolete paths Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): leave fused frame output repairs to separate PR Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: restore native contracts and regenerate config reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: slice nullable sparse frame fields after span selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): intersect anchored timestamp selection windows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): name anchored timestamp boundary case type Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): align partitioned pair checks with hybrid operands Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com>	2026-10-27 17:41:38 +08:00
Dennis Zhuang	ba8156be47	fix(promql): keep the right side's labels for a filled left side across tables ... Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 16:21:31 +08:00
Dennis Zhuang	3d5f99596c	chore: merge narthex into feat/narthex-cross-table ... Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 15:47:23 +08:00
discord9	70638a769f	feat(promql): add explicit local canonical merge operator (#9969) ... * feat(promql): add explicit local canonical merge operator Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): tighten canonical merge accounting and validation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): remove redundant merge tree whitelist Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): retain canonical merge output label reservations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): finish canonical merge review simplifications Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the parent barrier over canonical merge Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): assert the two-selector binary stays unfused Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-27 15:15:20 +08:00
Dennis Zhuang	7bcdec1a83	chore: merge narthex into feat/narthex-cross-table ... Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/extension_plan/fused_series.rs # src/promql/src/extension_plan/fused_series/tests.rs	2026-10-27 14:39:04 +08:00
Dennis Zhuang	ed562fc905	feat(promql): run vector-vector comparisons natively ... A one-to-one comparison between two selectors, such as `a > on(host) b` or `a == bool b`, kept the existing plan: the pair admission accepted only arithmetic operators. The kernel already pairs comparisons by the same matching rules and only differs in the value a pair keeps (the left sample when the comparison holds, or 1/0 with `bool`), so the operator check is dropped. This covers one table and two, and the point-wise wrappers around a pair, such as `(a > b) * 100`. Adds an end-to-end case over two tables comparing explicit rows, the legacy answer and the native placement for a filter and a `bool` comparison. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 12:36:20 +08:00
discord9	a9266b2da9	feat(promql): fuse independent operands of hybrid binary queries (#9973) ... I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## What's changed and what's your intention? Enable vmbench-lite 05/06/08 cross-metric arithmetic using two independently planned native operands and the existing DataFusion binary join, PromQL matching and arithmetic projection. Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) -> Fused B(M)` execute region-side; separate MergeScan branches feed the existing frontend Join. No native Join, second matching kernel or Exchange is added; same-table fused admission is preserved. A bounded analyzer pass locates each direct Fused(TableScan) through Alias/Projection/Filter/Window wrappers, rewrites only the fused subtree, and leaves all wrappers frontend-side. This preserves global uniqueness checks. Subquery bodies disable native admission and restore the prior flag before returning either success or error, avoiding use of outer evaluation bounds. ### Verification - `make fmt-check` - `cargo check --locked -p query -p tests-integration --all-targets` - Focused query admission/analyzer tests: 2 passed - `cargo nextest run --locked -p tests-integration --lib -E 'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on final source - `cargo clippy --locked -p query --all-targets --all-features -- -D warnings` - Independent source review approved. Standalone integration asserts two independent region-local Fused->SeriesScan branches, four nonempty partitions per input, frontend join placement, native/legacy complete-row parity, unmatched-series removal (05), outer scalar projection (06), and independent rate inputs (08). Additional cases cover subquery legacy fallback, valid reduced-label `on(job)` matching, frontend uniqueness Window plus `prom_assert_unique_match_group` Filter above MergeScan, and meaningful duplicate-match execution errors under both routes. ### Scope and limitations Standalone single-region per table, multiple scan partitions. Arithmetic and admitted one-to-one matching only. No distributed Join/Exchange, cross-region routing, group_left/right, set operators, native subqueries or multi-field expansion. No full workspace suite or performance benchmark is claimed. Based directly on narthex, independent of the global-TopK PR. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the matching labels.	2026-10-27 11:26:20 +08:00
Dennis Zhuang	c9423f0193	fix(promql): decline cross-table outputs the native node cannot name ... Under `or` the right table's tags join the left side's columns, and one of them can share a name with the left side's field or time index; building the output schema then failed instead of leaving the query to the existing plan. Such a union is now declined. The output-label walk also declines an aggregate or whole-vector node instead of passing its input's labels through, which would describe the wrong schema if one were ever reached. Adds an end-to-end case over two tables whose tags, field and time index are all named differently, partly flushed and partly in the memtable, under both series scan modes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 02:38:21 +08:00
Dennis Zhuang	4ce757b611	feat(promql): run cross-table selector pairs natively ... A binary or set operation between selectors of two tables, such as `rate(a[5m]) / rate(b[5m])`, fell back to the existing plan: the native route required every input to share one metric, time index and tag list, and a fused node is placed inside the one region it reads. Two tables are two regions, so such a pair now runs on the frontend. Each input becomes a `RawSeriesSource` that rides in its own merge scan: the region binds its scan to that table's metric, tags and fields, decodes the labels and answers one explicit-label raw frame stream, reading its scan partitions one after another so each series stays contiguous. The fused node consumes those frames directly. Pairs on one table keep running in the region; partitioned tables keep the existing plan. A point-wise wrapper around an admitted pair, such as `100 * (1 - a / b)`, is admitted too, on one table and across two. The output labels are now derived from the whole tree with each input's own tags, and the physical node takes them from the output schema instead of narrowing by the root alone. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-27 01:53:07 +08:00
discord9	6b7d1bca4c	test(promql): strengthen hybrid binary review regressions ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-27 01:03:31 +08:00
discord9	f882c1182d	refactor(promql): reuse prepared native operand context ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 19:02:38 +08:00
discord9	c25b6a9fbd	feat(promql): fuse independent operands of hybrid binary queries ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 17:05:42 +08:00
Dennis Zhuang	c24997e543	fix(promql): leave multi-field set operations to the legacy plan ... The legacy planner refuses `and`, `or` and `unless` over several fields (MultiFieldsNotSupported), and Prometheus has one value per sample, so per-field presence has no reference semantics. The compiler now declines a set operation unless each side has exactly one field, and the query keeps the legacy plan and its error. Errors raised by the shared two-operand runtime now name the set node instead of calling every such node "the binary node". Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-26 16:55:15 +08:00
Dennis Zhuang	cef469b450	fix(query): keep two-selector native plans off partitioned tables ... A two-scan FusedSeries over a table split into several regions has no placement: DistPlannerAnalyzer rejects it with "only accept plan with at most one child", so the query failed instead of falling back. This hit the existing arithmetic pair and, since set operators now compile, set operators too. Decline the native route for two selectors when the table has partition columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-26 16:28:53 +08:00
Dennis Zhuang	1be0df046c	feat(promql): compile and run set operators natively ... Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new `NativeExpr::Set` node instead of refusing them. The node runs the existing MATCH-SET kernel through the binary runtime, which retains both operands until the input ends, so a set operation reads every series of both sides in one execution. The set kernel now reports its workspace like the binary kernel does: kept rows, the filter side's match groups, the sort and the merge are charged to the node's reservation. The compiler refuses a fill value on a set operator, which Prometheus rejects and the parser accepts, and sides with different field counts. Same-table set operations over two scans take the existing two-scan native route. Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-10-26 16:23:05 +08:00
discord9	b20e22b426	refactor(series): unify raw and evaluated SeriesFrame codec ... SeriesFrame is the internal transport of the Narthex native PromQL path, and it could only carry raw samples: an evaluated result had to fall back to a plain row table. This makes one self-describing Arrow format carry both, so a receiver decodes a frame from the frame alone, without the scanner, region, or planning context its producer had. What is unified is the physical representation and its codec, not the runtime data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are; only the adapter boundaries branch on the sample shape. Three combinations are legal and every other one is refused in both directions: raw + key + timestamps a storage raw-series scan raw + explicit + timestamps a raw input whose labels are decoded evaluated + explicit + grid_spans a compute node's output An evaluated frame cannot claim a storage key, because the computation may rewrite its labels; raw samples have no grid to be addressed on. The schema builder and the declaration parser both enforce that, so a frame cannot be written or read in a shape no producer could mean. An evaluated frame declares its GridSpec once in schema metadata and carries absolute grid positions per row as spans, so a sparse grid costs its runs and not its holes: nothing is expanded, and no position a chunk does not hold is filled in. A chunk with no position is legal exactly when it closes its chain, and it is kept as one frame row, because dropping it would drop the finality it exists to carry. Layout is v2 and the writer and reader of the reserved greptime.series.* metadata live in one place each, so a producer and a consumer cannot spell one frame two ways. The unreleased v1 format is not read: the in-tree producers, consumers, tests, and the two frozen wire fixtures all move to v2 together. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit 176f6f6479)	2026-10-26 12:57:11 +08:00
discord9	9a70220974	test(promql): cover the multi-value timestamp refusal ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 11:25:44 +08:00
discord9	24e214b7d4	feat(promql): bind instant anchors and execute native timestamps ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 11:25:43 +08:00
discord9	3c409f91de	test(promql): align admission expectations with literal entry ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 00:23:44 +08:00
discord9	5bcd96cead	feat(promql): execute zero-input scalar roots through fused children ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-26 00:13:13 +08:00
discord9	9720e1bf88	style(promql): attach native predicate rustdoc directly ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 21:59:49 +08:00
discord9	1df0cb6af7	test(promql): admit complete aggregate in native shape coverage ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 21:46:57 +08:00
discord9	463d4a5c8f	fix(promql): align aggregate admission with boxed parser inputs ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 21:35:20 +08:00
discord9	73b9a77932	feat(promql): adapt complete aggregate wiring to recursive native runtime ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 21:25:05 +08:00
discord9	8dbf08d252	feat(promql): execute independent selector scans in fused binary trees ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 20:10:32 +08:00
discord9	b4c94cbd3a	test(promql): align native display and shared-source scan expectations ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:58:20 +08:00
discord9	4748081579	feat(promql): fan out a shared raw scan to bound binary inputs ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:47:42 +08:00
discord9	2f967ceb45	test(promql): correct distinct-input arithmetic expectations ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:15:15 +08:00
discord9	39fcdbdbcf	feat(promql): lower independently bound vector binary operands ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:14:35 +08:00
discord9	6b38352079	test(promql): avoid cloning a selector binding for validation ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:07:56 +08:00
discord9	b1e3b38a28	test(promql): cover selector bounds and recursive binding contracts ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 13:05:09 +08:00
discord9	3790693dd2	feat(promql): bind selector occurrences without node paths ... Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>	2026-10-25 12:58:22 +08:00