feat(promql): execute split native aggregates as partial state over a state router (#10147)

* feat(datatypes): add the shared aggregate-state representation

A frame that carries partial aggregate state has to say which aggregate it
is state of and which accumulator components it carries, or a consumer can
read a sum state as an average one. These types are that statement: the
kind, the ordered component list, one position's values, and the schema
that validates the two against each other.

Nothing uses them yet; the frame content that carries them follows.

Signed-off-by: discord9 <discord9@163.com>

* feat(datatypes): carry aggregate state as a third frame content

A state frame keeps the series-major columns a frame already has — the
key, finality, grid spans and labels — and puts each aggregate field's
accumulators in one list of component structs under `fields`, the shape a
classic histogram payload already uses. An absent list item is that field
having no contribution at that position, so a hole stays a hole instead of
becoming a zero, and a group state — which carries no component at all —
still says per field where it was folded.

The reader keeps typed handles into the batch rather than expanding the
state into rows and rebuilding it, so the merge above it reads the arrays
where they lie.

Signed-off-by: discord9 <discord9@163.com>

* feat(common-query): pack and unpack aggregate-state series frames

The packer takes what a partial aggregate produces — one group, its carried
positions and its per-field state — and the unpacker hands back the row with
typed handles into the batch. That asymmetry is deliberate: export
columnarises once, import must not rebuild the state to read it.

Validation mirrors the evaluated pair: the declaration is parsed, contiguity
required, the empty batch decoded before the stream is returned, and kind,
component order, grid and field count are each checked with the part that
disagreed named.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): run native partial and final aggregates

An aggregate node used to compute only `Complete`, refusing the stages a
split aggregate needs. It now folds evaluated chunks into kernel state for
`Partial` and merges that state into values for `Final`, so the compiler's
`Partial -> Final` plan has a runtime behind it.

The executor gains a third chunk kind: an input leaf accepts raw, evaluated
or state chunks according to what its binding produces, never guessing from
a schema, and a state chunk cannot be handed to a node that reads values.
The state binding now carries the aggregate kind and its ordered components,
derived from the kernel's own state fields, so a sum state cannot be merged
as an average one.

The sibling router refuses aggregate state explicitly for now; the node that
routes it follows in its own change.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): route aggregate state by its group labels

A state row is one group, complete as one producer wrote it: there is
nothing to chain and nothing to merge, so the existing repartition node —
whose invariants are all about assembling a key's chain and waiting for
every input to close — is the wrong place for it. This node does one thing:
send each row to the output its canonical labels name, so every
contribution of a group reaches one owner and the merge above sees all of
them.

The read belongs to the outputs that were claimed, not to the plan, so
dropping every output stops it and returns what it reserved.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): execute partial aggregates as state frames

A fused node now reads its inputs by what they produce — raw samples,
evaluated values, or aggregate state — instead of inferring the kind from a
schema, and a partial root answers state rather than rows: the executor's
state chunks are packed into the aggregate-state frame the merge above
consumes.

The temporary partial-fold bridge goes with it: no marker field, no side
channel fold, no compatibility accessor. A tree that folds on the side is
exactly what this change exists to remove, and the aggregate merge that
replaces it is a separate node.

The state path derives its mode and its binding from one reading of the
root, so the two cannot disagree, and no production path panics on a
malformed tree.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): split native aggregates into partial and final trees

The previous commit made a `Partial` aggregate export its state, but left
`AggregateFinalize` in place: it still called `FusedSeries::try_new_partial`
and `partial_fold`, which that commit deleted, so the crate did not compile.

Replace the bridge with the two trees that run a split aggregate:

- `native/split.rs` owns the cut. It moves out of `aggregate_finalize.rs`
  unchanged in structure, and `Split::into_trees` binds the two halves: the
  prefix under a `Partial` aggregate answering aggregate state, and a `Final`
  aggregate consuming that state with the suffix above it.
- The suffix is narrower than the old merge admitted. Only point-wise maps and
  a binary operation with a constant side may sit above the cut; `topk`, the
  classic-histogram functions, a second aggregate and a vector matching are
  declined, because the groups a `Final` aggregate owns are split across
  partitions and each of those reads a whole vector.
- `FusedSeries` gains `try_new_state_export` (a `Partial` node declares its
  own state frame schema rather than taking a caller's), `split_aggregate`
  (declining shapes become `None`, not an error), `reads_aggregate_state` and
  the state-input path of `to_execution_plan_with_parallelism_option`, which
  routes state through `StateRepartitionExec` and lets a cross-series tree read
  state from more than one partition.
- The runtime executes the `Final` stage: the guard that refused it is gone.

`AggregateFinalize`, `aggregate_exchange` and their tests are deleted; nothing
in the crate referenced the exchange module.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): route aggregate state through the series repartitioner

Replace the dedicated state-repartition node with the existing series
repartition execution: state batches are routed by their canonical labels
without being rebuilt as StateSeries, and each row slice is charged for the
Arrow wrappers it newly allocates while the parent buffers stay charged once.

Carry a producer's state charge with the chunk it hands over, so the exporter
that packs rows across polls remains accountable for the batch it still holds
instead of releasing it at handoff.

Read a Final aggregate's value and tag columns positionally, which is the
order the split's output schema is built in, and name them through the
legacy expression only at the outer boundary; a bare sum over a metric whose
value column is named after the whole expression no longer fails planning.

Pack aggregate-state frames in one pass over positions and validity instead
of scanning the contributing rows for every position.

Cover the changed behaviour with execution tests for bare global and grouped
sums over one- and two-value-field tables, for the handed-over state charge,
and for the export lifecycle on drain and early drop.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): bound tree depth on both sides and validate merge inputs

The depth limit was decode-only and set to a number nobody had measured. A
129-deep tree could therefore be built, serialized and shipped, while the
receiver refused it; and in CI the boundary test itself aborted with a stack
overflow, because a 2 MiB thread was asked to build, serialize, decode and
drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is
not covered by a guard that only wraps `Deserialize`.

Measured instead of guessed. A probe walked build, serialize, decode,
validate and drop at increasing depth inside a named 2 MiB thread: 32 through
128 completed, 160 aborted. The cap is now 64, half the highest passing depth,
and the same number is enforced in `NativeTree::try_new` and in the
deserializer. Verified through the real constructor: a 64-node tree
round-trips, and a 65-node tree is refused when it is built as well as when it
is decoded, so sender and receiver accept the same set. The boundary test runs
on a named thread, which is how a future failure will identify itself.

The merge node accepted any single input. `with_exprs_and_inputs` rebuilt
`AggregateFinalize` over whatever it was handed, on the reasoning that a
rewritten input only changes where the state comes from. Nothing checked that
it was state at all -- and until now nothing could produce a counterexample,
because a values-producing producer could not be serialized. The serializer
was standing in for a validation that belonged to the node. `with_exprs_and_inputs`
now rejects an input whose schema is not `state_schema(binding.tags)`, so
substituting a computed-values producer fails at the logical node with a
message naming the expected schema, rather than depending on an encoding
limitation. `AggregateFinalizeExec` already had the same check at physical
planning; the serializer stays permissive.

The three placement tests in `commutativity.rs` asserted the old refusal as
their mechanism -- an ordinary fused node "fails to encode", a partial
producer "cannot be pushed down". Those expectations were the encoding
limitation wearing a test's clothes. They now assert the responsibilities that
remain: an ordinary fused node round-trips through the encoder, a partial
producer rides to the region while its finalize stays on the frontend, and a
values producer handed to a merge is refused by the schema contract.

The test-only `NPRQ` envelope is gone. It framed trees with its own version, a
1 MiB cap and its own rejections, none of which production had, so
`native_tree_envelope_rejects_version_size_trailing_and_truncation` was
proving a helper's protocol while the PR description claimed a production
guarantee. Trees are tested through plain postcard, transport through the real
`FusedSeries` interface. The redundant clone-and-revalidate in
`FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already
routes through `try_new`.

Signed-off-by: discord9 <discord9@163.com>

* style: let taplo order the workspace dependency entry

`taplo format --check` runs over the whole workspace in CI, not over the two
manifests I had checked. It sorts the `[workspace.dependencies]` table, so
`postcard` belongs in alphabetical position inside the
external-dependencies block rather than immediately after its comment. Local
checking was narrower than the CI command, which is why this only surfaced
there.

Signed-off-by: discord9 <discord9@163.com>

* style: place postcard in the sorted position the review asked for

Inside the external-dependencies block, `taplo` was willing to leave the entry
before the `axum` comment, because a comment line ends a reorder group. The
review asked for the alphabetical position, which `taplo format --check` also
accepts, so use it.

Signed-off-by: discord9 <discord9@163.com>

* docs(promql): name the real reason a multi-source name grouping declines

The comments still said the fused node's declared columns cannot name
`__name__`. Since #10106 a discovered source declares it, so a nameless
selector's `sum by (__name__)` survives; what actually declines is a tree
reading more than one *named* source, where the table name is the metric
name and no source declares the tag. Same conditions, accurate reason.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): find the topk producer by behaviour, not by node count

`native_aggregate_topk_over_global_partitions` asserted that the whole plan
holds exactly one fused node. Once native frames cross the region boundary a
plan legitimately holds the frontend root and the region producers at the
same time, so that count is no longer an invariant.

The test now selects the segmented producer and the ranking node by what
their plans do: at least one producer contains `segment` and none of them
runs `topk`, and exactly one node across the fused and final captures ranks.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
discord9
2026-11-04 21:25:41 +08:00
committed by GitHub
parent fb533c4b9c
commit fe2e364d23
42 changed files with 10107 additions and 6358 deletions
+296 -2
View File
@@ -43,10 +43,11 @@ use common_recordbatch::{DfRecordBatch, DfSendableRecordBatchStream};
use datafusion_common::DataFusionError;
use datatypes::arrow::datatypes::{Schema as ArrowSchema, SchemaRef as ArrowSchemaRef};
use datatypes::error::InvalidSeriesSnafu;
use datatypes::series::aggregate_state::{AggregateStateSchema, StateSeries};
use datatypes::series::frame::{
FrameContent, FrameLabelMode, LABEL_DECODE_VERSION, LabelDecodeColumn, LabelDecodeEncoding,
LabelDecodeSpec, LabelDecodeTag, SeriesFrameBuilder, SeriesFrameCursor, SeriesFrameDeclaration,
SeriesFrameDecoder, series_frame_label_decode, series_frame_schema,
SeriesFrameDecoder, StateFrameRow, series_frame_label_decode, series_frame_schema,
};
use datatypes::series::raw::RawSeriesChunk;
use datatypes::series::samples::{Series, SeriesSchema};
@@ -71,6 +72,18 @@ const ORDERING_KEY: &str = "greptime.series.ordering";
pub type SendableEvaluatedSeriesStream =
Pin<Box<dyn Stream<Item = Result<(Series, bool), BoxedError>> + Send>>;
/// A stream of aggregate-state chunks and their producer-local finality.
///
/// Export accepts the existing field-major [`StateSeries`] representation and columnarises each
/// chunk once. Import is intentionally asymmetric: [`SendableStateFrameStream`] retains typed Arrow
/// handles so consumers do not rebuild a nested state tree.
pub type SendableStateSeriesStream =
Pin<Box<dyn Stream<Item = Result<(StateSeries, bool), BoxedError>> + Send>>;
/// A stream of decoded state rows retaining typed handles into their Arrow batch.
pub type SendableStateFrameStream =
Pin<Box<dyn Stream<Item = Result<StateFrameRow, BoxedError>> + Send>>;
/// Builds the canonical raw frame schema with the producer's label binding and ordering.
///
/// Ordering records a producer guarantee; it is not inferred from frame contents. `Interleaved`
@@ -282,6 +295,124 @@ pub fn pack_evaluated_series(
}))
}
/// Packs aggregate-state chunks into bounded DataFusion record batches.
///
/// The producer supplies one [`StateSeries`] per group chunk; the cursor columnarises it into the
/// frame once. Aggregate state requires a grid, explicit labels and contiguous ordering. The state
/// schema is revalidated with its own constructor before any stream is returned.
pub fn pack_aggregate_state(
source: SendableStateSeriesStream,
state: Arc<AggregateStateSchema>,
schema: Arc<SeriesSchema>,
) -> datafusion_common::Result<DfSendableRecordBatchStream> {
if schema.eval_grid.is_none() {
return Err(invalid("an aggregate state frame needs an evaluation grid"));
}
AggregateStateSchema::try_new(state.kind, state.components.to_vec()).map_err(codec_error)?;
let frame_schema = datatypes::series::frame::aggregate_state_frame_schema(
&state,
&schema,
FrameLabelMode::Explicit,
)
.map_err(codec_error)?;
let mut metadata = frame_schema.metadata().clone();
metadata.insert(ORDERING_KEY.to_string(), "contiguous".to_string());
let frame_schema = Arc::new(ArrowSchema::new_with_metadata(
frame_schema.fields().clone(),
metadata,
));
let builder = SeriesFrameBuilder::try_new_aggregate_state(
&state,
&schema,
FrameLabelMode::Explicit,
SeriesFrameBuilder::DEFAULT_MAX_ROWS,
SeriesFrameBuilder::DEFAULT_MAX_BYTES,
)
.map_err(codec_error)?;
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(&state, &schema, FrameLabelMode::Explicit)
.map_err(codec_error)?;
// Decoding the empty batch validates the exact declared layout before publishing this stream.
decoder
.decode_aggregate_state(&DfRecordBatch::new_empty(frame_schema.clone()))
.map_err(codec_error)?;
Ok(Box::pin(PackStateStream {
source: Some(source),
state,
schema,
batches: PackBatches::new(builder, frame_schema),
}))
}
/// Unpacks aggregate-state frames while retaining Arrow-backed state rows.
///
/// Kind and ordered components must both agree: the kind describes the accumulator semantics while
/// components describe its physical values. Expected fields and grid are checked independently.
pub fn unpack_aggregate_state(
source: DfSendableRecordBatchStream,
expected_state: Option<Arc<AggregateStateSchema>>,
expected_series: Option<Arc<SeriesSchema>>,
) -> datafusion_common::Result<SendableStateFrameStream> {
let frame_schema = source.schema();
let declaration = SeriesFrameDeclaration::parse(frame_schema.as_ref()).map_err(codec_error)?;
if declaration.content() != FrameContent::AggregateState {
return Err(invalid(format!(
"expected an aggregate state frame, found {}",
declaration.content().as_str()
)));
}
ensure_contiguous_ordering(series_frame_ordering(frame_schema.as_ref())?)?;
let state = declaration
.aggregate_state()
.ok_or_else(|| invalid("aggregate state declaration has no aggregate state"))?;
if let Some(expected) = expected_state {
if state.kind != expected.kind {
return Err(invalid(
"aggregate state kind does not match expected state",
));
}
if state.components != expected.components {
return Err(invalid(
"aggregate state components do not match expected state",
));
}
}
let schema = Arc::new(declaration.series().clone());
if let Some(expected) = expected_series {
if expected.fields.len() != schema.fields.len()
|| expected
.fields
.iter()
.zip(schema.fields.iter())
.any(|(a, b)| a.name != b.name)
{
return Err(invalid(
"aggregate state fields do not match expected series",
));
}
if expected.eval_grid.as_ref().map(|grid| grid.spec())
!= schema.eval_grid.as_ref().map(|grid| grid.spec())
{
return Err(invalid(
"aggregate state grid does not match expected series",
));
}
}
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(state, &schema, FrameLabelMode::Explicit)
.map_err(codec_error)?;
// Validate even empty streams before returning them, rather than deferring declaration errors.
decoder
.decode_aggregate_state(&DfRecordBatch::new_empty(frame_schema.clone()))
.map_err(codec_error)?;
Ok(Box::pin(UnpackStateStream {
source: Some(source),
frame_schema,
pending: Vec::new().into_iter(),
decoder,
}))
}
/// Unpacks DataFusion batches declared by [`pack_raw_series`].
///
/// The frame is the sole source of decode information: the payload fields, the label mode, the
@@ -357,7 +488,10 @@ pub fn unpack_evaluated_series(
let frame_schema = source.schema();
let declaration = SeriesFrameDeclaration::parse(frame_schema.as_ref()).map_err(codec_error)?;
if declaration.content() != FrameContent::Evaluated {
return Err(invalid("expected an evaluated frame, found a raw one"));
return Err(invalid(format!(
"expected an evaluated frame, found {}",
declaration.content().as_str()
)));
}
ensure_contiguous_ordering(series_frame_ordering(frame_schema.as_ref())?)?;
let schema = Arc::new(declaration.series().clone());
@@ -809,6 +943,166 @@ impl datafusion::physical_plan::RecordBatchStream for PackStream {
}
}
/// The pack stream of [`pack_aggregate_state`].
struct PackStateStream {
source: Option<SendableStateSeriesStream>,
state: Arc<AggregateStateSchema>,
schema: Arc<SeriesSchema>,
batches: PackBatches,
}
impl PackStateStream {
fn finish(&mut self) {
self.source = None;
self.batches.release();
}
}
impl Stream for PackStateStream {
type Item = datafusion_common::Result<DfRecordBatch>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = &mut *self;
loop {
match this.batches.advance() {
Ok(Some(batch)) => return this.batches.declare(batch),
Ok(None) => {}
Err(error) => {
this.finish();
return Poll::Ready(Some(Err(codec_error(error))));
}
}
let polled = match this.source.as_mut() {
Some(source) => source.as_mut().poll_next(cx),
None => return Poll::Ready(None),
};
match polled {
Poll::Pending => {
return match this.batches.flush() {
Ok(Some(batch)) => this.batches.declare(batch),
Ok(None) => Poll::Pending,
Err(error) => {
this.finish();
Poll::Ready(Some(Err(codec_error(error))))
}
};
}
Poll::Ready(None) => {
let flushed = this.batches.flush();
let declared = this.batches.frame_schema.clone();
this.finish();
return match flushed {
Ok(Some(batch)) => {
match DfRecordBatch::try_new(declared, batch.columns().to_vec()) {
Ok(batch) => Poll::Ready(Some(Ok(batch))),
Err(error) => Poll::Ready(Some(Err(DataFusionError::from(error)))),
}
}
Ok(None) => Poll::Ready(None),
Err(error) => Poll::Ready(Some(Err(codec_error(error)))),
};
}
Poll::Ready(Some(Err(error))) => {
this.finish();
return Poll::Ready(Some(Err(upstream_error(error))));
}
Poll::Ready(Some(Ok((series, is_last)))) => {
match SeriesFrameCursor::try_new_aggregate_state(
&this.state,
&this.schema,
series,
is_last,
) {
Ok(cursor) => this.batches.install(cursor),
Err(error) => {
this.finish();
return Poll::Ready(Some(Err(codec_error(error))));
}
}
}
}
}
}
}
impl datafusion::physical_plan::RecordBatchStream for PackStateStream {
fn schema(&self) -> ArrowSchemaRef {
self.batches.frame_schema.clone()
}
}
/// The unpack stream of [`unpack_aggregate_state`].
struct UnpackStateStream {
source: Option<DfSendableRecordBatchStream>,
frame_schema: ArrowSchemaRef,
pending: std::vec::IntoIter<StateFrameRow>,
decoder: SeriesFrameDecoder,
}
impl UnpackStateStream {
fn finish(&mut self) {
self.source = None;
self.pending = Vec::new().into_iter();
}
}
impl Stream for UnpackStateStream {
type Item = Result<StateFrameRow, BoxedError>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
let this = &mut *self;
let mut empty_batches = 0;
loop {
if let Some(row) = this.pending.next() {
return Poll::Ready(Some(Ok(row)));
}
let polled = match this.source.as_mut() {
Some(source) => source.as_mut().poll_next(cx),
None => return Poll::Ready(None),
};
match polled {
Poll::Pending => return Poll::Pending,
Poll::Ready(None) => {
this.finish();
return Poll::Ready(None);
}
Poll::Ready(Some(Err(error))) => {
this.finish();
return Poll::Ready(Some(Err(BoxedError::new(
crate::error::Error::GeneralDataFusion {
error,
location: snafu::Location::default(),
},
))));
}
Poll::Ready(Some(Ok(batch))) => {
let result = if batch.schema() != this.frame_schema {
Err(invalid("aggregate state frame schema mismatch"))
} else {
this.decoder
.decode_aggregate_state(&batch)
.map_err(codec_error)
};
match result {
Ok(rows) if rows.is_empty() => {
empty_batches += 1;
if empty_batches == MAX_EMPTY_BATCHES_PER_POLL {
cx.waker().wake_by_ref();
return Poll::Pending;
}
}
Ok(rows) => this.pending = rows.into_iter(),
Err(error) => {
this.finish();
return Poll::Ready(Some(Err(unpack_error(error))));
}
}
}
}
}
}
}
/// The pack stream of [`pack_evaluated_series`].
struct PackEvaluatedStream {
source: Option<SendableEvaluatedSeriesStream>,
@@ -14,6 +14,7 @@
mod evaluated;
mod lifecycle;
mod state;
use std::collections::{HashMap, VecDeque};
use std::io::Cursor;
@@ -0,0 +1,391 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::io::Cursor;
use std::sync::Arc;
use common_error::ext::BoxedError;
use common_recordbatch::DfSendableRecordBatchStream;
use datatypes::arrow::compute::concat_batches;
use datatypes::arrow::ipc::{reader::StreamReader, writer::StreamWriter};
use datatypes::arrow::record_batch::RecordBatch;
use datatypes::series::aggregate_state::{
AggregateStateKind, AggregateStateSchema, AggregateStateValues, StateComponent, StateSeries,
};
use datatypes::series::frame::{FrameLabelMode, StateFrameRow};
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use datatypes::series::time::{GridSpan, TimeAxis};
use datatypes::series::{Label, SeriesLabels};
use futures_util::StreamExt;
use super::super::{
SendableStateSeriesStream, pack_aggregate_state, pack_evaluated_series, unpack_aggregate_state,
unpack_evaluated_series,
};
use super::{df_source, evaluated_grid, evaluated_schema};
fn group() -> SeriesLabels {
SeriesLabels::try_new(
Some(Arc::from("cpu")),
vec![Label {
name: Arc::from("host"),
value: Arc::from("a"),
}],
)
.unwrap()
}
fn state_schema(kind: AggregateStateKind) -> Arc<AggregateStateSchema> {
let components = match kind {
AggregateStateKind::Group => vec![],
AggregateStateKind::Sum => vec![StateComponent::Value, StateComponent::Compensation],
AggregateStateKind::Count => vec![StateComponent::Count],
AggregateStateKind::Avg => vec![
StateComponent::Value,
StateComponent::Mean,
StateComponent::Compensation,
StateComponent::Count,
StateComponent::IncrementalMean,
],
AggregateStateKind::Min | AggregateStateKind::Max => vec![StateComponent::Value],
AggregateStateKind::Stddev | AggregateStateKind::Stdvar => vec![
StateComponent::Value,
StateComponent::Mean,
StateComponent::Count,
],
};
Arc::new(AggregateStateSchema::try_new(kind, components).unwrap())
}
fn series_schema(fields: &[&str]) -> Arc<SeriesSchema> {
Arc::new(SeriesSchema {
fields: fields
.iter()
.map(|name| FieldSpec {
name: Arc::from(*name),
})
.collect(),
eval_grid: Some(evaluated_grid()),
})
}
fn wrong_grid_schema() -> Arc<SeriesSchema> {
Arc::new(SeriesSchema {
fields: vec![FieldSpec {
name: Arc::from("a"),
}]
.into(),
eval_grid: Some(Arc::new(
datatypes::series::time::EvalGrid::try_new(
datatypes::series::time::GridSpec::try_new(1, 1000, 512).unwrap(),
)
.unwrap(),
)),
})
}
fn input(fields: Vec<Vec<Option<AggregateStateValues>>>) -> (StateSeries, bool) {
let time = TimeAxis::from_grid_spans(evaluated_grid(), vec![GridSpan::new(1, 3)]).unwrap();
(
StateSeries {
metric: group(),
time,
states: fields,
},
true,
)
}
fn source(items: Vec<(StateSeries, bool)>) -> SendableStateSeriesStream {
Box::pin(futures_util::stream::iter(
items.into_iter().map(Ok::<_, BoxedError>),
))
}
async fn packed(
rows: Vec<(StateSeries, bool)>,
state: Arc<AggregateStateSchema>,
schema: Arc<SeriesSchema>,
) -> Vec<RecordBatch> {
let mut stream = pack_aggregate_state(source(rows), state, schema).unwrap();
let mut batches = Vec::new();
while let Some(batch) = stream.next().await {
batches.push(batch.unwrap());
}
batches
}
fn unpack_source(
batches: Vec<RecordBatch>,
schema: Arc<datatypes::arrow::datatypes::Schema>,
) -> DfSendableRecordBatchStream {
df_source(batches, schema).0
}
async fn rows(
batches: Vec<RecordBatch>,
state: Option<Arc<AggregateStateSchema>>,
schema: Option<Arc<SeriesSchema>>,
) -> Vec<StateFrameRow> {
let frame = batches
.first()
.map(|batch| batch.schema())
.unwrap_or_else(|| {
datatypes::series::frame::aggregate_state_frame_schema(
state.as_ref().unwrap(),
schema.as_ref().unwrap(),
FrameLabelMode::Explicit,
)
.unwrap()
});
let mut stream = unpack_aggregate_state(unpack_source(batches, frame), state, schema).unwrap();
let mut rows = Vec::new();
while let Some(row) = stream.next().await {
rows.push(row.unwrap());
}
rows
}
fn values(value: f64) -> Option<AggregateStateValues> {
Some(AggregateStateValues {
value,
compensation: value / 10.0,
..Default::default()
})
}
fn assert_state_row(row: &StateFrameRow, expected: &[f64], last: bool) {
assert_eq!(row.metric(), &group());
assert!(row.is_last() == last);
assert_eq!(
row.time()
.positions()
.map(|(position, _)| position)
.collect::<Vec<_>>(),
vec![1, 2, 3]
);
let fields = row.fields().unwrap();
for (position, value) in expected.iter().enumerate() {
assert!(fields.is_present(0, position).unwrap());
assert_eq!(
fields
.field_values(0, StateComponent::Value)
.unwrap()
.unwrap()
.value(position),
*value
);
}
}
fn assert_group_row(row: &StateFrameRow, last: bool) {
assert_eq!(row.metric(), &group());
assert_eq!(row.is_last(), last);
assert_eq!(
row.time()
.positions()
.map(|(position, _)| position)
.collect::<Vec<_>>(),
vec![1, 2, 3]
);
let fields = row.fields().unwrap();
for field in 0..fields.len() {
for position in 0..3 {
assert!(fields.is_present(field, position).unwrap());
}
}
}
#[tokio::test]
async fn aggregate_state_roundtrip_preserves_components_and_group_rows() {
let sum = state_schema(AggregateStateKind::Sum);
let schema = series_schema(&["a"]);
let input_rows = vec![input(vec![vec![values(1.0), values(2.0), values(3.0)]])];
let batches = packed(input_rows, sum.clone(), schema.clone()).await;
let decoded = rows(batches, Some(sum), Some(schema)).await;
assert_eq!(decoded.len(), 1);
assert_state_row(&decoded[0], &[1.0, 2.0, 3.0], true);
let group_state = state_schema(AggregateStateKind::Group);
let group_schema = series_schema(&["a"]);
let group_batches = packed(
vec![input(vec![vec![values(0.0), values(0.0), values(0.0)]])],
group_state.clone(),
group_schema.clone(),
)
.await;
let group_rows = rows(group_batches, Some(group_state), Some(group_schema)).await;
assert_eq!(group_rows.len(), 1);
assert_group_row(&group_rows[0], true);
}
#[tokio::test]
async fn group_presence_is_per_field() {
let state = state_schema(AggregateStateKind::Group);
let schema = series_schema(&["a", "b"]);
let batches = packed(
vec![input(vec![
vec![values(0.0), None, values(0.0)],
vec![None, values(0.0), None],
])],
state.clone(),
schema.clone(),
)
.await;
let decoded = rows(batches, Some(state), Some(schema)).await;
let fields = decoded[0].fields().unwrap();
assert_eq!(
(0..3)
.map(|i| fields.is_present(0, i).unwrap())
.collect::<Vec<_>>(),
vec![true, false, true]
);
assert_eq!(
(0..3)
.map(|i| fields.is_present(1, i).unwrap())
.collect::<Vec<_>>(),
vec![false, true, false]
);
}
#[tokio::test]
async fn aggregate_state_ipc_concat_roundtrip() {
let state = state_schema(AggregateStateKind::Sum);
let schema = series_schema(&["a"]);
let batches = packed(
vec![input(vec![vec![values(1.0), values(2.0), values(3.0)]])],
state.clone(),
schema.clone(),
)
.await;
let expected = rows(batches.clone(), Some(state.clone()), Some(schema.clone())).await;
let mut bytes = Cursor::new(Vec::new());
{
let mut writer = StreamWriter::try_new(&mut bytes, batches[0].schema().as_ref()).unwrap();
for batch in &batches {
writer.write(batch).unwrap();
}
writer.finish().unwrap();
}
bytes.set_position(0);
let reader = StreamReader::try_new(bytes, None).unwrap();
let read = reader.map(Result::unwrap).collect::<Vec<_>>();
let concatenated = concat_batches(&batches[0].schema(), &read).unwrap();
let decoded = rows(vec![concatenated], Some(state), Some(schema)).await;
assert_eq!(decoded.len(), expected.len());
assert_state_row(&decoded[0], &[1.0, 2.0, 3.0], true);
}
#[tokio::test]
async fn aggregate_state_refusals_name_mismatched_parts() {
let state = state_schema(AggregateStateKind::Sum);
let schema = series_schema(&["a"]);
let batches = packed(
vec![input(vec![vec![values(1.0), values(2.0), values(3.0)]])],
state.clone(),
schema.clone(),
)
.await;
for (expected, expected_schema, part) in [
(
Some(state_schema(AggregateStateKind::Count)),
Some(schema.clone()),
"kind",
),
(
Some(Arc::new(AggregateStateSchema {
kind: AggregateStateKind::Sum,
components: vec![StateComponent::Compensation, StateComponent::Value].into(),
})),
Some(schema.clone()),
"components",
),
(Some(state.clone()), Some(wrong_grid_schema()), "grid"),
(
Some(state.clone()),
Some(series_schema(&["a", "b"])),
"fields",
),
] {
let frame = batches[0].schema();
let (source, _, _) = df_source(batches.clone(), frame);
let error = unpack_aggregate_state(source, expected, expected_schema)
.err()
.unwrap()
.to_string();
assert!(
error.to_lowercase().contains(part),
"{error} did not name {part}"
);
}
}
#[tokio::test]
async fn aggregate_state_rejects_evaluated_and_evaluated_rejects_state() {
let state = state_schema(AggregateStateKind::Sum);
let schema = series_schema(&["a"]);
let batches = packed(
vec![input(vec![vec![values(1.0), values(2.0), values(3.0)]])],
state,
schema,
)
.await;
let eval =
pack_evaluated_series(Box::pin(futures_util::stream::empty()), evaluated_schema()).unwrap();
let frame = eval.schema();
let (source, _, _) = df_source(vec![], frame.clone());
assert!(
unpack_aggregate_state(source, None, None)
.err()
.unwrap()
.to_string()
.contains("evaluated")
);
let (source, _, _) = df_source(batches.clone(), batches[0].schema());
assert!(
unpack_evaluated_series(source, None)
.err()
.unwrap()
.to_string()
.contains("aggregate_state")
);
let eval =
pack_evaluated_series(Box::pin(futures_util::stream::empty()), evaluated_schema()).unwrap();
let frame = eval.schema();
let (source, _, _) = df_source(vec![], frame);
assert!(
unpack_aggregate_state(source, None, None)
.err()
.unwrap()
.to_string()
.contains("evaluated")
);
}
#[tokio::test]
async fn aggregate_state_empty_stream_terminates() {
let state = state_schema(AggregateStateKind::Sum);
let schema = series_schema(&["a"]);
let batches = packed(vec![], state.clone(), schema.clone()).await;
assert!(batches.is_empty());
let frame = datatypes::series::frame::aggregate_state_frame_schema(
&state,
&schema,
datatypes::series::frame::FrameLabelMode::Explicit,
)
.unwrap();
let (source, _, _) = df_source(vec![], frame);
let mut decoded = unpack_aggregate_state(source, Some(state), Some(schema)).unwrap();
assert!(decoded.next().await.is_none());
}
+1
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub mod aggregate_state;
pub mod frame;
pub mod histogram;
pub mod raw;
+328
View File
@@ -0,0 +1,328 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use arrow::array::{Array, BooleanArray, Float64Array, ListArray, StructArray};
use arrow::datatypes::DataType;
use crate::error::{InvalidSeriesSnafu, Result};
use crate::series::SeriesLabels;
use crate::series::time::TimeAxis;
/// Aggregate whose accumulator is represented by a state frame.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
pub enum AggregateStateKind {
Sum,
Count,
Avg,
Min,
Max,
Group,
Stddev,
Stdvar,
}
impl AggregateStateKind {
/// Canonical metadata spelling.
pub const fn as_str(self) -> &'static str {
match self {
Self::Sum => "sum",
Self::Count => "count",
Self::Avg => "avg",
Self::Min => "min",
Self::Max => "max",
Self::Group => "group",
Self::Stddev => "stddev",
Self::Stdvar => "stdvar",
}
}
/// Parses a canonical aggregate name without treating unknown kinds as another state.
pub fn parse(text: &str) -> Result<Self> {
match text {
"sum" => Ok(Self::Sum),
"count" => Ok(Self::Count),
"avg" => Ok(Self::Avg),
"min" => Ok(Self::Min),
"max" => Ok(Self::Max),
"group" => Ok(Self::Group),
"stddev" => Ok(Self::Stddev),
"stdvar" => Ok(Self::Stdvar),
_ => invalid("unknown aggregate state kind"),
}
}
fn components(self) -> &'static [StateComponent] {
use StateComponent as C;
match self {
Self::Sum => &[C::Value, C::Compensation],
Self::Count => &[C::Count],
Self::Avg => &[
C::Value,
C::Mean,
C::Compensation,
C::Count,
C::IncrementalMean,
],
Self::Min | Self::Max => &[C::Value],
Self::Group => &[],
Self::Stddev | Self::Stdvar => &[C::Value, C::Mean, C::Count],
}
}
}
/// One named accumulator column.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
pub enum StateComponent {
Value,
Mean,
Compensation,
Count,
IncrementalMean,
}
impl StateComponent {
/// Canonical metadata spelling.
pub const fn as_str(self) -> &'static str {
match self {
Self::Value => "value",
Self::Mean => "mean",
Self::Compensation => "compensation",
Self::Count => "count",
Self::IncrementalMean => "incremental_mean",
}
}
/// Parses a canonical component name, refusing unknown wire values.
pub fn parse(text: &str) -> Result<Self> {
match text {
"value" => Ok(Self::Value),
"mean" => Ok(Self::Mean),
"compensation" => Ok(Self::Compensation),
"count" => Ok(Self::Count),
"incremental_mean" => Ok(Self::IncrementalMean),
_ => invalid("unknown aggregate state component"),
}
}
}
/// Values for all accumulator components at one field and grid position.
#[derive(Clone, Copy, Debug, Default, PartialEq)]
pub struct AggregateStateValues {
/// Accumulator value.
pub value: f64,
/// Running mean.
pub mean: f64,
/// Compensation retained for stable summation.
pub compensation: f64,
/// Observation count represented as a floating-point accumulator.
pub count: f64,
/// Whether the mean is incremental.
pub incremental_mean: bool,
}
/// Validated aggregate kind and ordered component layout.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AggregateStateSchema {
pub kind: AggregateStateKind,
pub components: Arc<[StateComponent]>,
}
impl AggregateStateSchema {
/// Creates a schema only when its ordered components exactly describe its kind.
pub fn try_new(kind: AggregateStateKind, components: Vec<StateComponent>) -> Result<Self> {
if kind != AggregateStateKind::Group && components.is_empty() {
return invalid("aggregate state components must not be empty");
}
if components.as_slice() != kind.components() {
return invalid("aggregate state components do not match kind");
}
Ok(Self {
kind,
components: components.into(),
})
}
/// Parses the aggregate kind and ordered comma-separated component metadata.
pub fn from_text(kind: &str, components: &str) -> Result<Self> {
let kind = AggregateStateKind::parse(kind)?;
let parsed = if components.is_empty() {
Vec::new()
} else {
components
.split(',')
.map(StateComponent::parse)
.collect::<Result<Vec<_>>>()?
};
Self::try_new(kind, parsed)
}
/// Returns the canonical ordered component metadata text.
pub fn components_text(&self) -> String {
self.components
.iter()
.map(|component| component.as_str())
.collect::<Vec<_>>()
.join(",")
}
}
/// A group-labelled sparse aggregate-state chunk on a shared time axis.
#[derive(Clone, Debug)]
pub struct StateSeries {
/// Complete group identity.
pub metric: SeriesLabels,
/// Grid positions carried by this chunk.
pub time: TimeAxis,
/// Field-major states; `None` denotes a field position without a contribution.
pub states: Vec<Vec<Option<AggregateStateValues>>>,
}
fn invalid<T>(reason: &str) -> Result<T> {
InvalidSeriesSnafu {
reason: reason.to_string(),
}
.fail()
}
/// Columnar state fields retained in Arrow form so consumers can read component values in place.
/// Null items are field-local holes, not zero-valued states.
#[derive(Clone, Debug)]
pub struct StateFieldsBatch {
kind: AggregateStateKind,
components: Arc<[StateComponent]>,
fields: Vec<StructArray>,
}
impl StateFieldsBatch {
/// Validates the ordered component layout once and keeps the row's typed Arrow arrays.
pub fn try_from_arrow(fields: StructArray, schema: &AggregateStateSchema) -> Result<Self> {
if fields.num_columns() == 0 {
return invalid("state fields array has no fields");
}
let mut items = Vec::with_capacity(fields.num_columns());
for (column, field) in fields.columns().iter().zip(fields.fields()) {
let DataType::List(item) = field.data_type() else {
return invalid("state field is not a list");
};
let DataType::Struct(members) = item.data_type() else {
return invalid("state list item is not a struct");
};
if members.len() != schema.components.len() {
return invalid("state component count mismatch");
}
for (member, expected) in members.iter().zip(schema.components.iter()) {
if member.name() != expected.as_str() {
return invalid("state component order mismatch");
}
let ty = if *expected == StateComponent::IncrementalMean {
DataType::Boolean
} else {
DataType::Float64
};
if member.data_type() != &ty {
return invalid("state component type mismatch");
}
}
let list = column
.as_any()
.downcast_ref::<ListArray>()
.ok_or_else(|| invalid_series("state field list type mismatch"))?;
if list.is_null(0) {
return invalid("null state field list");
}
let item_array = list.value(0);
let item_struct = item_array
.as_any()
.downcast_ref::<StructArray>()
.ok_or_else(|| invalid_series("state item struct type mismatch"))?;
items.push(item_struct.clone());
}
Ok(Self {
kind: schema.kind,
components: schema.components.clone(),
fields: items,
})
}
/// Number of logical series fields in this row.
pub fn len(&self) -> usize {
self.fields.len()
}
/// Whether this wrapper contains no series fields.
pub fn is_empty(&self) -> bool {
self.fields.is_empty()
}
/// The item struct for one field's list, sliced to the decoded row.
pub fn field(&self, index: usize) -> Option<&StructArray> {
self.fields.get(index)
}
/// Aggregate kind represented by these columns.
pub fn kind(&self) -> AggregateStateKind {
self.kind
}
/// Declared accumulator components in wire order.
pub fn components(&self) -> &[StateComponent] {
&self.components
}
/// Returns a numeric component array, or `None` if the component is not declared.
pub fn field_values(
&self,
field: usize,
component: StateComponent,
) -> Result<Option<&Float64Array>> {
let Some(index) = self.components.iter().position(|value| *value == component) else {
return Ok(None);
};
self.fields
.get(field)
.ok_or_else(|| invalid_series("state field index out of range"))?
.column(index)
.as_any()
.downcast_ref::<Float64Array>()
.map(Some)
.ok_or_else(|| invalid_series("state float component type mismatch"))
}
/// Returns a Boolean component array, or `None` if the component is not declared.
pub fn field_bool_values(
&self,
field: usize,
component: StateComponent,
) -> Result<Option<&BooleanArray>> {
let Some(index) = self.components.iter().position(|value| *value == component) else {
return Ok(None);
};
self.fields
.get(field)
.ok_or_else(|| invalid_series("state field index out of range"))?
.column(index)
.as_any()
.downcast_ref::<BooleanArray>()
.map(Some)
.ok_or_else(|| invalid_series("state boolean component type mismatch"))
}
/// Reads this field's item validity at one carried position; sibling fields never fill holes.
pub fn is_present(&self, field: usize, position: usize) -> Result<bool> {
let items = self
.fields
.get(field)
.ok_or_else(|| invalid_series("state field index out of range"))?;
if position >= items.len() {
return invalid("state position out of range");
}
Ok(!items.is_null(position))
}
}
fn invalid_series(reason: &str) -> crate::error::Error {
InvalidSeriesSnafu {
reason: reason.to_string(),
}
.build()
}
File diff suppressed because it is too large Load Diff
+784 -1
View File
@@ -28,6 +28,7 @@ use crate::series::histogram::{BucketSideView, CounterResetHint, HistogramBatchB
use crate::series::samples::{
DenseSamplesRef, FieldSpec, SampleRef, Samples, SeriesFields, SeriesSchema,
};
use crate::series::time::GridRange;
use crate::series::{Label, SeriesLabels};
fn test_schema(names: &[&str]) -> SeriesSchema {
@@ -1358,7 +1359,7 @@ fn evaluated_incremental_measure_matches_full_measure_at_exact_byte_budget() {
metric: Some(metric.clone()),
is_last: true,
time: RowTime::Spans(vec![GridSpan::new(0, 8), GridSpan::new(10, 1)]),
samples: vec![samples.clone()],
payload: RowPayload::Samples(vec![samples.clone()]),
bytes: 0,
};
let exact_budget = measure_row(&whole).unwrap();
@@ -4673,3 +4674,785 @@ fn the_frozen_text_covers_every_datatype_variant() {
"one fixture entry per datatype variant, no duplicates"
);
}
#[cfg(test)]
mod aggregate_state_acceptance {
use super::*;
use crate::series::aggregate_state::{
AggregateStateKind as Kind, AggregateStateSchema, AggregateStateValues as Values,
StateComponent as Component, StateSeries,
};
fn state_schema(fields: &[&str], len: usize) -> SeriesSchema {
let base = test_schema(fields);
SeriesSchema {
fields: base.fields,
eval_grid: Some(Arc::new(
crate::series::time::EvalGrid::try_new(
crate::series::time::GridSpec::try_new(0, 1, len as u64).unwrap(),
)
.unwrap(),
)),
}
}
fn state_frame(
kind: Kind,
schema: &SeriesSchema,
spans: Vec<GridSpan>,
states: Vec<Vec<Option<Values>>>,
last: bool,
) -> (RecordBatch, SeriesFrameDecoder) {
let components = match kind {
Kind::Sum => vec![Component::Value, Component::Compensation],
Kind::Count => vec![Component::Count],
Kind::Avg => vec![
Component::Value,
Component::Mean,
Component::Compensation,
Component::Count,
Component::IncrementalMean,
],
Kind::Min | Kind::Max => vec![Component::Value],
Kind::Group => vec![],
Kind::Stddev | Kind::Stdvar => {
vec![Component::Value, Component::Mean, Component::Count]
}
};
let state = AggregateStateSchema::try_new(kind, components).unwrap();
let time = TimeAxis::from_grid_spans(schema.eval_grid.clone().unwrap(), spans).unwrap();
let cursor = SeriesFrameCursor::try_new_aggregate_state(
&state,
schema,
StateSeries {
metric: metric(Some("m"), &[("job", "x")]),
time,
states,
},
last,
)
.unwrap();
let mut builder = SeriesFrameBuilder::try_new_aggregate_state(
&state,
schema,
FrameLabelMode::Explicit,
8,
1 << 20,
)
.unwrap();
let mut cursor = cursor;
assert!(cursor.append_to(&mut builder).unwrap().is_none());
let batch = builder.flush().unwrap().unwrap();
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(&state, schema, FrameLabelMode::Explicit)
.unwrap();
(batch, decoder)
}
fn values(n: usize, seed: f64) -> Vec<Option<Values>> {
(0..n)
.map(|i| {
Some(Values {
value: seed + i as f64,
mean: seed + 10.0 + i as f64,
compensation: seed + 20.0 + i as f64,
count: seed + 30.0 + i as f64,
incremental_mean: i % 2 == 0,
})
})
.collect()
}
#[test]
fn group_state_keeps_field_local_presence_through_ipc_and_slices() {
let schema = state_schema(&["a", "b"], 6);
let (batch, decoder) = state_frame(
Kind::Group,
&schema,
vec![GridSpan::new(0, 4)],
vec![
vec![Some(Values::default()), None, Some(Values::default()), None],
vec![None; 4],
],
true,
);
let expected = decoder.decode_aggregate_state(&batch).unwrap();
let baseline_metric = expected[0].metric().clone();
let baseline_positions = expected[0].time().positions().collect::<Vec<_>>();
let baseline_last = expected[0].is_last();
let check = |rows: Vec<StateFrameRow>| {
let row = &rows[0];
let fields = row.fields().unwrap();
let presence = |field| {
(0..4)
.map(|i| fields.is_present(field, i).unwrap())
.collect::<Vec<_>>()
};
assert_eq!(presence(0), vec![true, false, true, false]);
assert_eq!(presence(1), vec![false; 4]);
assert_eq!(row.metric(), &baseline_metric);
assert_eq!(
row.time().positions().collect::<Vec<_>>(),
baseline_positions
);
assert_eq!(row.is_last(), baseline_last);
};
check(expected);
let mut ipc = Vec::new();
{
let mut writer = StreamWriter::try_new(&mut ipc, &batch.schema()).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
}
let batches = StreamReader::try_new(std::io::Cursor::new(ipc), None)
.unwrap()
.map(|batch| batch.unwrap())
.collect::<Vec<_>>();
let joined = concat_batches(&batch.schema(), &batches).unwrap();
check(decoder.decode_aggregate_state(&joined).unwrap());
check(decoder.decode_aggregate_state(&joined.slice(0, 1)).unwrap());
}
#[test]
fn aggregate_state_kinds_round_trip_values_presence_labels_axis_and_finality() {
let kinds = [
Kind::Sum,
Kind::Count,
Kind::Avg,
Kind::Min,
Kind::Max,
Kind::Group,
Kind::Stddev,
Kind::Stdvar,
];
for kind in kinds {
let schema = state_schema(&["x", "y"], 5);
let (batch, decoder) = state_frame(
kind,
&schema,
vec![GridSpan::new(1, 2), GridSpan::new(4, 1)],
vec![
values(3, 1.0),
vec![
None,
Some(values(3, 1.0)[1].unwrap()),
Some(values(3, 1.0)[2].unwrap()),
],
],
true,
);
let rows = decoder.decode_aggregate_state(&batch).unwrap();
assert_eq!(rows.len(), 1);
let row = &rows[0];
assert_eq!(row.metric(), &metric(Some("m"), &[("job", "x")]));
assert_eq!(
row.time().positions().map(|(p, _)| p).collect::<Vec<_>>(),
vec![1, 2, 4]
);
assert!(row.is_last());
let fields = row.fields().unwrap();
assert_eq!(fields.kind(), kind);
assert_eq!(
(0..3)
.map(|i| fields.is_present(0, i).unwrap())
.collect::<Vec<_>>(),
vec![true; 3]
);
assert_eq!(
(0..3)
.map(|i| fields.is_present(1, i).unwrap())
.collect::<Vec<_>>(),
vec![false, true, true]
);
for field in 0..2 {
for (component, expected) in [
(Component::Value, [1.0, 2.0, 3.0]),
(Component::Mean, [11.0, 12.0, 13.0]),
(Component::Compensation, [21.0, 22.0, 23.0]),
(Component::Count, [31.0, 32.0, 33.0]),
] {
if let Some(array) = fields.field_values(field, component).unwrap() {
for (i, value) in expected.iter().enumerate() {
if fields.is_present(field, i).unwrap() {
assert_eq!(array.value(i), *value);
}
}
}
}
}
}
}
#[test]
fn aggregate_state_all_kinds_keep_values_and_presence_through_ipc_slicing_and_concat() {
let kinds = [
Kind::Sum,
Kind::Count,
Kind::Avg,
Kind::Min,
Kind::Max,
Kind::Group,
Kind::Stddev,
Kind::Stdvar,
];
let components = [
Component::Value,
Component::Mean,
Component::Compensation,
Component::Count,
Component::IncrementalMean,
];
for kind in kinds {
let schema = state_schema(&["x", "y"], 5);
let source_values = values(4, 3.0);
let states = || {
vec![
source_values.clone(),
vec![
None,
source_values[1],
Some(values(4, 30.0)[2].unwrap()),
None,
],
]
};
// Two rows, so slicing and re-concatenating at an offset has something to shift: with
// one row, a decoder that ignored the offset would still agree with the baseline.
let (first, decoder) = state_frame(
kind,
&schema,
vec![GridSpan::new(0, 3), GridSpan::new(4, 1)],
states(),
false,
);
let (second, _) = state_frame(
kind,
&schema,
vec![GridSpan::new(0, 3), GridSpan::new(4, 1)],
states(),
true,
);
let batch = concat_batches(&first.schema(), &[first, second]).unwrap();
let baseline = decoder.decode_aggregate_state(&batch).unwrap();
assert_eq!(baseline.len(), 2);
// This migrates #10080's property: IPC bytes plus batch concatenation must not change
// field-local presence, accumulator values, labels, axis, or finality.
let mut ipc = Vec::new();
{
let mut writer = StreamWriter::try_new(&mut ipc, &batch.schema()).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
}
let read_back = StreamReader::try_new(std::io::Cursor::new(ipc), None)
.unwrap()
.map(|batch| batch.unwrap())
.collect::<Vec<_>>();
assert_eq!(read_back.len(), 1);
let ipc_batch = &read_back[0];
assert_eq!(ipc_batch.num_rows(), baseline.len());
// Both halves carry a non-zero offset after concatenation, so the decoder has to honour
// the child arrays' offsets rather than reading from their first element.
let sliced = vec![ipc_batch.slice(1, 1), ipc_batch.slice(0, 1)];
let joined = concat_batches(&batch.schema(), &sliced).unwrap();
assert_eq!(
joined.num_rows(),
baseline.len(),
"{kind:?}: every row survives slicing and concatenation"
);
let check = |rows: &[StateFrameRow], expected: &[&StateFrameRow], context: &str| {
assert_eq!(rows.len(), expected.len(), "{context}: row count");
for (row, expected) in rows.iter().zip(expected) {
let expected: &StateFrameRow = expected;
assert_eq!(row.metric(), expected.metric(), "{context}: labels");
assert_eq!(row.is_last(), expected.is_last(), "{context}: finality");
assert_eq!(
row.time().positions().collect::<Vec<_>>(),
expected.time().positions().collect::<Vec<_>>(),
"{context}: positions"
);
let actual_fields = row.fields().unwrap();
let expected_fields = expected.fields().unwrap();
assert_eq!(
actual_fields.kind(),
expected_fields.kind(),
"{context}: kind"
);
assert_eq!(
actual_fields.components(),
expected_fields.components(),
"{context}: components"
);
if kind == Kind::Avg {
// Pin the full set once, so iterating a candidate list cannot silently
// skip every component a kind does not have.
assert_eq!(
actual_fields.components(),
&components,
"{context}: av components"
);
}
for field in 0..expected_fields.len() {
let positions = expected_fields
.field(field)
.expect("a declared field has an item array")
.len();
for position in 0..positions {
assert_eq!(
actual_fields.is_present(field, position).unwrap(),
expected_fields.is_present(field, position).unwrap(),
"{context}: presence {kind:?} field {field} position {position}"
);
}
for component in components {
// Only the incremental mean is boolean; the rest are float64. Asking
// for the wrong one is itself an error, so the branch is by component.
if component == Component::IncrementalMean {
let expected_values =
expected_fields.field_bool_values(field, component).unwrap();
let actual_values =
actual_fields.field_bool_values(field, component).unwrap();
match (expected_values, actual_values) {
(Some(expected_values), Some(actual_values)) => {
for position in 0..expected_values.len() {
if expected_fields.is_present(field, position).unwrap()
{
assert_eq!(
actual_values.value(position),
expected_values.value(position),
"{context}: {kind:?} {component:?} \
field {field} position {position}"
);
}
}
}
(None, None) => {}
_ => panic!(
"{context}: {kind:?} {component:?} component changed"
),
}
continue;
}
let expected_values =
expected_fields.field_values(field, component).unwrap();
let actual_values =
actual_fields.field_values(field, component).unwrap();
match (expected_values, actual_values) {
(Some(expected_values), Some(actual_values)) => {
for position in 0..expected_values.len() {
if expected_fields.is_present(field, position).unwrap() {
assert_eq!(
actual_values.value(position),
expected_values.value(position),
"{context}: {kind:?} {component:?} field {field} \
position {position}"
);
}
}
}
(None, None) => {}
_ => panic!("{context}: {kind:?} {component:?} component changed"),
}
}
}
}
};
// The slices are concatenated in reverse row order on purpose: the decoded order must
// follow the concatenation, not the batch the rows came from.
check(
&decoder.decode_aggregate_state(&joined).unwrap(),
&[&baseline[1], &baseline[0]],
"ipc+concat",
);
// A slice of the joined batch: the decoder must honour the offset rather than reading
// the first rows of the underlying arrays.
let tail = joined.slice(1, joined.num_rows() - 1);
check(
&decoder.decode_aggregate_state(&tail).unwrap(),
&[&baseline[0]],
"sliced",
);
}
}
#[test]
fn aggregate_state_cursor_moves_dense_states_without_filtering() {
let schema = state_schema(&["a", "b"], 4);
let state = AggregateStateSchema::try_new(
Kind::Avg,
vec![
Component::Value,
Component::Mean,
Component::Compensation,
Component::Count,
Component::IncrementalMean,
],
)
.unwrap();
let states = vec![values(4, 1.0), values(4, 5.0)];
let allocation = states[0].as_ptr();
// A grid axis rather than a span axis, so the test can tell the chunk's own axis from a
// rebuilt one.
let time =
TimeAxis::Grid(GridRange::try_new(schema.eval_grid.clone().unwrap(), 0, 4).unwrap());
let cursor = SeriesFrameCursor::try_new_aggregate_state(
&state,
&schema,
StateSeries {
metric: metric(Some("m"), &[]),
time,
states,
},
true,
)
.unwrap();
let ChunkWalk::State(walk) = cursor.walk else {
panic!("expected state walk")
};
// The dense fast path preserves the caller's field allocation and values verbatim.
assert_eq!(walk.series.states[0].as_ptr(), allocation);
assert_eq!(walk.series.states, vec![values(4, 1.0), values(4, 5.0)]);
assert_eq!(
walk.series
.time
.positions()
.map(|(p, _)| p)
.collect::<Vec<_>>(),
vec![0, 1, 2, 3]
);
// And it keeps the axis it was handed. A rebuilt axis always states spans, so a grid axis
// surviving here is what says the dense case did not rebuild one.
assert!(
matches!(walk.series.time, TimeAxis::Grid(_)),
"a dense chunk kept its own axis: {:?}",
walk.series.time
);
}
/// Every component of one field is absent exactly where that field is, so they share one mask.
///
/// A position is carried when any field has a state there, so a hole needs a second field: the
/// two fields below cover each other's positions and each is absent where the other is present.
#[test]
fn state_components_share_their_fields_presence_mask() {
let schema = state_schema(&["x", "y"], 4);
let values = values(3, 1.0);
let (batch, decoder) = state_frame(
Kind::Avg,
&schema,
vec![GridSpan::new(0, 3)],
vec![
vec![Some(values[0].unwrap()), None, Some(values[2].unwrap())],
vec![None, Some(values[1].unwrap()), None],
],
true,
);
let rows = decoder.decode_aggregate_state(&batch).unwrap();
assert_eq!(rows.len(), 1);
let fields = rows[0].fields().unwrap();
assert_eq!(fields.len(), 2);
let mut masks = Vec::new();
// Field 0 is present at {0,2} and field 1 at {1}, so their masks differ in both content and
// hole count.
for (field, holes) in [1usize, 2].into_iter().enumerate() {
for component in [
Component::Value,
Component::Mean,
Component::Compensation,
Component::Count,
] {
let array = fields.field_values(field, component).unwrap().unwrap();
let nulls = array.nulls().expect("a hole in this field");
assert_eq!(nulls.null_count(), holes, "field {field} {component:?}");
masks.push(nulls.inner().inner().as_ptr());
}
let boolean = fields
.field_bool_values(field, Component::IncrementalMean)
.unwrap()
.expect("avg declares the incremental mean");
masks.push(
boolean
.nulls()
.expect("the same hole")
.inner()
.inner()
.as_ptr(),
);
}
// One mask per field, shared by that field's components: five components of field 0 agree,
// then five of field 1.
assert!(
masks[..5].windows(2).all(|pair| pair[0] == pair[1])
&& masks[5..].windows(2).all(|pair| pair[0] == pair[1]),
"one mask per field, shared by its components: {masks:?}"
);
assert_ne!(
masks[0], masks[5],
"two fields with different holes are two masks"
);
}
/// A field with no hole states no mask at all, which is what a null buffer builder produces.
#[test]
fn a_field_without_holes_carries_no_null_buffer() {
let schema = state_schema(&["x"], 3);
let (batch, decoder) = state_frame(
Kind::Sum,
&schema,
vec![GridSpan::new(0, 3)],
vec![values(3, 1.0)],
true,
);
let rows = decoder.decode_aggregate_state(&batch).unwrap();
let fields = rows[0].fields().unwrap();
let array = fields
.field_values(0, Component::Value)
.unwrap()
.expect("sum declares the value");
assert!(array.nulls().is_none());
let mean = fields
.field_bool_values(0, Component::IncrementalMean)
.unwrap();
assert!(mean.is_none(), "sum declares no incremental mean");
}
#[test]
fn aggregate_state_cursor_compacts_sparse_fields_and_preserves_runs() {
let schema = state_schema(&["a", "b"], 7);
let state = AggregateStateSchema::try_new(
Kind::Sum,
vec![Component::Value, Component::Compensation],
)
.unwrap();
let a = values(4, 1.0);
let b = values(4, 5.0);
let input = vec![
vec![Some(a[0].unwrap()), None, Some(a[2].unwrap()), None],
vec![None, Some(b[1].unwrap()), None, Some(b[3].unwrap())],
];
let time = TimeAxis::from_grid_spans(
schema.eval_grid.clone().unwrap(),
vec![
GridSpan::new(0, 1),
GridSpan::new(2, 1),
GridSpan::new(4, 1),
GridSpan::new(6, 1),
],
)
.unwrap();
let cursor = SeriesFrameCursor::try_new_aggregate_state(
&state,
&schema,
StateSeries {
metric: metric(Some("m"), &[]),
time,
states: input,
},
false,
)
.unwrap();
let ChunkWalk::State(walk) = cursor.walk else {
panic!("expected state walk")
};
assert_eq!(
walk.series
.time
.positions()
.map(|(p, _)| p)
.collect::<Vec<_>>(),
vec![0, 2, 4, 6]
);
assert_eq!(
walk.runs
.iter()
.map(|run| (run.position, run.len))
.collect::<Vec<_>>(),
vec![(0, 1), (2, 1), (4, 1), (6, 1)]
);
assert_eq!(
walk.series.states[0],
vec![Some(a[0].unwrap()), None, Some(a[2].unwrap()), None]
);
assert_eq!(
walk.series.states[1],
vec![None, Some(b[1].unwrap()), None, Some(b[3].unwrap())]
);
}
#[test]
fn sparse_state_runs_and_empty_batch_decode() {
let schema = state_schema(&["x"], 10);
let (batch, decoder) = state_frame(
Kind::Sum,
&schema,
vec![GridSpan::new(0, 3), GridSpan::new(5, 3)],
vec![values(6, 0.0)],
true,
);
let rows = decoder.decode_aggregate_state(&batch).unwrap();
assert_eq!(
rows[0]
.time()
.runs()
.map(|r| (r.position, r.len))
.collect::<Vec<_>>(),
vec![(0, 3), (5, 3)]
);
assert!(
decoder
.decode_aggregate_state(&batch.slice(0, 0))
.unwrap()
.is_empty()
);
}
}
#[cfg(test)]
mod aggregate_state_decoder_rejections {
use super::*;
use crate::series::aggregate_state::{
AggregateStateKind as Kind, AggregateStateSchema, AggregateStateValues as Values,
StateComponent as Component, StateSeries,
};
fn fixture() -> (RecordBatch, SeriesFrameDecoder) {
let base = test_schema(&["x"]);
let schema = SeriesSchema {
fields: base.fields,
eval_grid: Some(Arc::new(
crate::series::time::EvalGrid::try_new(
crate::series::time::GridSpec::try_new(0, 1, 3).unwrap(),
)
.unwrap(),
)),
};
let state = AggregateStateSchema::try_new(
Kind::Sum,
vec![Component::Value, Component::Compensation],
)
.unwrap();
let time =
TimeAxis::from_grid_spans(schema.eval_grid.clone().unwrap(), vec![GridSpan::new(0, 3)])
.unwrap();
let cursor = SeriesFrameCursor::try_new_aggregate_state(
&state,
&schema,
StateSeries {
metric: metric(Some("m"), &[]),
time,
states: vec![vec![
Some(Values {
value: 1.0,
compensation: 2.0,
..Values::default()
});
3
]],
},
true,
)
.unwrap();
let mut cursor = cursor;
let mut builder = SeriesFrameBuilder::try_new_aggregate_state(
&state,
&schema,
FrameLabelMode::Explicit,
4,
4096,
)
.unwrap();
assert!(cursor.append_to(&mut builder).unwrap().is_none());
let batch = builder.flush().unwrap().unwrap();
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(&state, &schema, FrameLabelMode::Explicit)
.unwrap();
(batch, decoder)
}
fn replace_field(batch: &RecordBatch, replacement: ArrayRef) -> RecordBatch {
let old = batch
.column(3)
.as_any()
.downcast_ref::<StructArray>()
.unwrap();
let mut columns = old.columns().to_vec();
columns[0] = replacement;
let fields = Arc::new(
StructArray::try_new(old.fields().clone(), columns, old.nulls().cloned()).unwrap(),
);
let mut batch_columns = batch.columns().to_vec();
batch_columns[3] = fields;
RecordBatch::try_new(batch.schema(), batch_columns).unwrap()
}
#[test]
fn present_state_item_with_missing_component_is_rejected() {
let (batch, decoder) = fixture();
let list = batch
.column(3)
.as_any()
.downcast_ref::<StructArray>()
.unwrap()
.column(0)
.as_any()
.downcast_ref::<ListArray>()
.unwrap();
let item = list
.values()
.as_any()
.downcast_ref::<StructArray>()
.unwrap();
let mut columns = item.columns().to_vec();
columns[0] = Arc::new(Float64Array::from(vec![None, Some(1.0), Some(1.0)]));
let broken_item = Arc::new(
StructArray::try_new(item.fields().clone(), columns, item.nulls().cloned()).unwrap(),
);
let broken_list = Arc::new(
ListArray::try_new(
match list.data_type() {
DataType::List(f) => f.clone(),
_ => unreachable!(),
},
list.offsets().clone(),
broken_item,
list.nulls().cloned(),
)
.unwrap(),
);
let broken = replace_field(&batch, broken_list);
assert!(decoder.decode_aggregate_state(&broken).is_err());
}
#[test]
fn state_list_length_must_match_carried_positions() {
let (batch, decoder) = fixture();
let list = batch
.column(3)
.as_any()
.downcast_ref::<StructArray>()
.unwrap()
.column(0)
.as_any()
.downcast_ref::<ListArray>()
.unwrap();
let broken = Arc::new(
ListArray::try_new(
match list.data_type() {
DataType::List(f) => f.clone(),
_ => unreachable!(),
},
OffsetBuffer::new(ScalarBuffer::from(vec![0i32, 2])),
list.values().slice(0, 2),
None,
)
.unwrap(),
);
assert!(
decoder
.decode_aggregate_state(&replace_field(&batch, broken))
.is_err()
);
}
}
+2 -4
View File
@@ -16,13 +16,11 @@
//! is keyed by absolute grid index, never by local row number; sparse state and trimmed output
//! do not expand the full grid.
mod state;
pub(crate) mod state;
#[cfg(test)]
mod tests;
pub use state::{
AccumulatorState, FloatMerger, PartialAggregate, PartialGroup, StateField, merge_count_values,
};
pub use state::{AccumulatorState, FloatMerger, PartialAggregate, StateField, merge_count_values};
use std::cmp::Reverse;
use std::collections::hash_map::Entry;
+439 -87
View File
@@ -31,9 +31,14 @@ use std::collections::{BTreeMap, HashMap};
use std::mem::size_of;
use std::sync::Arc;
use datafusion::arrow::array::Array;
use datafusion::error::{DataFusionError, Result};
use datatypes::series::SeriesLabels;
use datatypes::series::aggregate_state::{
AggregateStateKind, AggregateStateValues, StateComponent, StateFieldsBatch, StateSeries,
};
use datatypes::series::samples::{SampleRef, Series, SeriesSchema};
use datatypes::series::time::{GridSpan, TimeAxis};
use crate::aggregate::{
AggregateOp, BLOCK_POINTS, Counts, FloatAggregator, FloatState, Group, LabelKey,
@@ -75,89 +80,120 @@ impl AggregateOp {
}
/// The state of one field of one group at one grid point.
#[derive(Clone, Copy, Debug, Default, PartialEq)]
pub struct AccumulatorState {
pub value: f64,
pub mean: f64,
pub compensation: f64,
pub count: f64,
pub incremental_mean: bool,
}
/// The state of one group: its points in ascending order, each with one state per field, `None`
/// where no sample of that field was folded in.
#[derive(Clone, Debug)]
pub struct PartialGroup {
pub metric: SeriesLabels,
pub points: Vec<(usize, Vec<Option<AccumulatorState>>)>,
}
///
/// The fold and the frame carry the same five components, so they share one type: the frame's own
/// [`AggregateStateValues`] is the definition, and this is the name the aggregate side reads it by.
pub use datatypes::series::aggregate_state::AggregateStateValues as AccumulatorState;
/// The state of one execution partition, with its groups in the order they first appeared.
///
/// A group is the frame's own [`StateSeries`]: field-major states addressed on that group's sparse
/// absolute grid positions. Exporting therefore hands the same representation over instead of
/// transposing a second one into it.
#[derive(Clone, Debug)]
pub struct PartialAggregate {
pub op: AggregateOp,
pub groups: Vec<PartialGroup>,
pub groups: Vec<StateSeries>,
}
/// One Arrow-backed aggregate state row, independent of executor plumbing.
#[derive(Clone, Debug)]
pub(crate) struct StateRow {
pub metric: SeriesLabels,
pub time: TimeAxis,
pub fields: StateFieldsBatch,
pub kind: AggregateStateKind,
}
impl FloatAggregator {
/// The state folded so far, for a merge with the state of other partitions.
pub fn finish_partial(self) -> PartialAggregate {
pub fn finish_partial(self) -> Result<PartialAggregate> {
self.finish_partial_inner()
}
/// [`Self::finish_partial`] that admits what it builds first: the fold state and the group,
/// point and state vectors, at the exact capacity [`Self::finish_partial_inner`] gives them. A
/// refusal consumes the aggregator, so the caller drops the execution.
/// [`Self::finish_partial`] that admits what it builds first: the fold state and the groups it
/// is exported as, at the exact capacity [`Self::finish_partial_inner`] gives them. A refusal
/// consumes the aggregator, so the caller drops the execution.
pub(crate) fn finish_partial_budgeted(
self,
mut budget: impl FnMut(usize) -> Result<()>,
) -> Result<PartialAggregate> {
let width = self.schema.fields.len();
let mut partial_bytes = self.groups.len().saturating_mul(size_of::<PartialGroup>());
let mut partial_bytes = self.groups.len().saturating_mul(size_of::<StateSeries>());
for group in &self.groups {
let points = seen_points(group, width);
partial_bytes = partial_bytes
.saturating_add(points.saturating_mul(size_of::<Point>()))
.saturating_add(size_of::<TimeAxis>())
.saturating_add(
points.saturating_mul(size_of::<GridSpan>().saturating_add(size_of::<usize>())),
)
.saturating_add(
width.saturating_mul(size_of::<Vec<Option<AggregateStateValues>>>()),
)
.saturating_add(
points
.saturating_mul(width)
.saturating_mul(size_of::<Option<AccumulatorState>>()),
.saturating_mul(size_of::<Option<AggregateStateValues>>()),
);
}
budget(self.state_bytes().saturating_add(partial_bytes))?;
Ok(self.finish_partial_inner())
self.finish_partial_inner()
}
fn finish_partial_inner(self) -> PartialAggregate {
/// Turns the fold into the frame's own groups: field-major states on sparse absolute positions.
///
/// The blocks are walked once, in ascending position order, so a position's states are appended
/// to every field at the same time and adjacent positions coalesce into one span as they come.
/// What comes out is the representation the frame writes; no per-point vector is built.
fn finish_partial_inner(self) -> Result<PartialAggregate> {
let width = self.schema.fields.len();
let mut groups: Vec<PartialGroup> = Vec::with_capacity(self.groups.len());
let grid = self.schema.eval_grid.as_ref().ok_or_else(|| {
DataFusionError::Execution("aggregate state is indexed on an evaluated grid".into())
})?;
let grid_len = grid.spec().len as usize;
let mut groups: Vec<StateSeries> = Vec::with_capacity(self.groups.len());
for group in self.groups {
let mut points: Vec<Point> = Vec::with_capacity(seen_points(&group, width));
for (&block, states) in &group.blocks {
for (offset, point) in states.chunks_exact(width).enumerate() {
if point.iter().any(|state| state.seen) {
points.push((
block * BLOCK_POINTS + offset,
point.iter().map(|state| state.partial()).collect(),
));
let points = seen_points(&group, width);
let mut spans: Vec<GridSpan> = Vec::with_capacity(points.min(1));
let mut states: Vec<Vec<Option<AggregateStateValues>>> =
(0..width).map(|_| Vec::with_capacity(points)).collect();
for (&block, block_states) in &group.blocks {
for (offset, point) in block_states.chunks_exact(width).enumerate() {
if !point.iter().any(|state| state.seen) {
continue;
}
let position = block * BLOCK_POINTS + offset;
if position >= grid_len {
return Err(DataFusionError::Execution(format!(
"aggregate state point {position} exceeds a grid of {grid_len}"
)));
}
match spans.last_mut() {
Some(span) if span.start() + span.len() == position => {
*span = GridSpan::new(span.start(), span.len() + 1);
}
_ => spans.push(GridSpan::new(position, 1)),
}
for (field, state) in point.iter().enumerate() {
states[field].push(state.partial());
}
}
}
groups.push(PartialGroup {
let time = TimeAxis::from_grid_spans(Arc::clone(grid), spans)
.map_err(|error| DataFusionError::Execution(error.to_string()))?;
groups.push(StateSeries {
metric: group.metric,
points,
time,
states,
});
}
PartialAggregate {
Ok(PartialAggregate {
op: self.op,
groups,
}
})
}
}
/// One point of a partial group: its grid position and one state per field.
type Point = (usize, Vec<Option<AccumulatorState>>);
/// The points of `group` that carry a state: the ones a partial of it holds.
fn seen_points(group: &Group, width: usize) -> usize {
group.blocks.values().fold(0usize, |count, states| {
@@ -170,16 +206,33 @@ fn seen_points(group: &Group, width: usize) -> usize {
})
}
/// Bytes one partial group keeps: its labels and its point and state vectors at the capacity they
/// were built with, so spare capacity is paid for rather than skipped.
fn group_bytes(group: &PartialGroup) -> usize {
metric_bytes(&group.metric)
.saturating_add(group.points.capacity().saturating_mul(size_of::<Point>()))
.saturating_add(group.points.iter().fold(0usize, |bytes, (_, states)| {
/// Bytes one exported group keeps: its labels and the field vectors, at the capacity they were
/// built with, so spare capacity is paid for rather than skipped.
fn group_bytes(group: &StateSeries) -> usize {
metric_bytes(&group.metric).saturating_add(group_field_bytes(group))
}
/// Bytes a group's axis and field vectors hold, without its labels.
pub(crate) fn group_field_bytes(group: &StateSeries) -> usize {
size_of::<TimeAxis>()
.saturating_add(
group
.time
.runs()
.count()
.saturating_mul(size_of::<GridSpan>().saturating_add(size_of::<usize>())),
)
.saturating_add(
group
.states
.capacity()
.saturating_mul(size_of::<Vec<Option<AggregateStateValues>>>()),
)
.saturating_add(group.states.iter().fold(0usize, |bytes, field| {
bytes.saturating_add(
states
field
.capacity()
.saturating_mul(size_of::<Option<AccumulatorState>>()),
.saturating_mul(size_of::<Option<AggregateStateValues>>()),
)
}))
}
@@ -277,23 +330,33 @@ impl FloatMerger {
)));
}
for group in &partial.groups {
for (point, states) in &group.points {
if *point >= points || states.len() != width {
return Err(DataFusionError::Execution(
"aggregate state does not match the grid or fields of the merge".into(),
));
}
let carried = group.time.len();
if group.states.len() != width
|| group.states.iter().any(|field| field.len() != carried)
{
return Err(DataFusionError::Execution(
"aggregate state does not match the grid or fields of the merge".into(),
));
}
if group.time.positions().any(|(point, _)| point >= points) {
return Err(DataFusionError::Execution(
"aggregate state does not match the grid or fields of the merge".into(),
));
}
}
let input_live = partial.groups.iter().fold(
partial
.groups
.capacity()
.saturating_mul(size_of::<PartialGroup>()),
.saturating_mul(size_of::<StateSeries>()),
|bytes, group| bytes.saturating_add(group_bytes(group)),
);
for group in partial.groups {
let metric = group.metric;
let StateSeries {
metric,
time,
states,
} = group;
// `label_key` copies the metric name into its own `Arc`, so the copy is admitted
// before it is made. A group that is already merged needs a key too, and its
// admission comes before anything else this call can allocate.
@@ -308,7 +371,95 @@ impl FloatMerger {
Some(index) => index,
None => self.open_group(metric, key, live, &mut budget)?,
};
self.merge_points(index, group.points, live, &mut budget)?;
self.merge_series(index, &time, &states, live, &mut budget)?;
}
Ok(())
}
/// Merges one Arrow-backed state row directly, without materializing a partial aggregate.
pub(crate) fn merge_row(
&mut self,
row: &StateRow,
mut budget: impl FnMut(usize) -> Result<()>,
) -> Result<()> {
let op = crate::native::state::op_of_state(row.kind);
if op != self.op {
return Err(DataFusionError::Execution(format!(
"cannot merge {:?} state into {:?}",
op, self.op
)));
}
let width = self.schema.fields.len();
if row.fields.len() != width {
return Err(DataFusionError::Execution(
"aggregate state does not match the grid or fields of the merge".into(),
));
}
let grid_len = self
.schema
.eval_grid
.as_ref()
.expect("checked at construction")
.spec()
.len as usize;
let mut opened = None;
let key_workspace = key_workspace_bytes(&row.metric);
let mut live = None;
for (point, position) in row.time.positions() {
if point >= grid_len {
return Err(DataFusionError::Execution(
"aggregate state does not match the grid or fields of the merge".into(),
));
}
let mut present = false;
for field in 0..width {
present |= row
.fields
.is_present(field, position)
.map_err(crate::grid_series::external)?;
}
if !present {
return Err(DataFusionError::Execution(
"aggregate state names a position with no contribution".into(),
));
}
let index = match opened {
Some(index) => index,
None => {
budget(self.retained.saturating_add(key_workspace))?;
let key = label_key(&row.metric);
let live_bytes = key_bytes(&key);
let index = match self.group_index.get(&key).copied() {
Some(index) => index,
None => {
self.open_group(row.metric.clone(), key, live_bytes, &mut budget)?
}
};
live = Some(live_bytes);
opened = Some(index);
index
}
};
let mut field = 0;
let mut state_error = None;
let states = std::iter::from_fn(|| {
if field == width || state_error.is_some() {
return None;
}
let current = field;
field += 1;
match state_at(&row.fields, current, position) {
Ok(state) => Some(state),
Err(error) => {
state_error = Some(error);
None
}
}
});
self.write_point(index, point, states, live.unwrap_or(0), &mut budget)?;
if let Some(error) = state_error {
return Err(error);
}
}
Ok(())
}
@@ -381,10 +532,29 @@ impl FloatMerger {
/// Adds the points of one partial group to group `index`. A missing block is admitted with its
/// states and B-tree node before either is allocated; an existing one only merges state.
fn merge_points(
fn merge_series(
&mut self,
index: usize,
points: Vec<Point>,
time: &TimeAxis,
states: &[Vec<Option<AggregateStateValues>>],
live: usize,
budget: &mut impl FnMut(usize) -> Result<()>,
) -> Result<()> {
// The axis carries the absolute positions and the row each one sits at, so a group's states
// are read field by field at that row: this is the same order the point-major form had.
for (point, row) in time.positions() {
let point_states = states.iter().map(|field| field[row]);
self.write_point(index, point, point_states, live, budget)?;
}
Ok(())
}
/// Writes one point's states, admitting a missing block before allocating it.
fn write_point(
&mut self,
index: usize,
point: usize,
states: impl Iterator<Item = Option<AccumulatorState>>,
live: usize,
budget: &mut impl FnMut(usize) -> Result<()>,
) -> Result<()> {
@@ -393,33 +563,29 @@ impl FloatMerger {
.saturating_mul(width)
.saturating_mul(size_of::<FloatState>());
let blocks = &mut self.groups[index].blocks;
for (point, states) in points {
let block_index = point / BLOCK_POINTS;
let entries = blocks.len();
let block = match blocks.entry(block_index) {
BTreeEntry::Occupied(entry) => entry.into_mut(),
BTreeEntry::Vacant(entry) => {
budget(
self.retained
.saturating_add(live)
.saturating_add(block_bytes)
.saturating_add(btree_backing_bytes(entries.saturating_add(1))),
)?;
self.retained = self
.retained
let block_index = point / BLOCK_POINTS;
let entries = blocks.len();
let block = match blocks.entry(block_index) {
BTreeEntry::Occupied(entry) => entry.into_mut(),
BTreeEntry::Vacant(entry) => {
budget(
self.retained
.saturating_add(live)
.saturating_add(block_bytes)
.saturating_add(btree_backing_bytes(entries.saturating_add(1)))
.saturating_sub(btree_backing_bytes(entries));
entry.insert(
vec![FloatState::default(); BLOCK_POINTS * width].into_boxed_slice(),
)
}
};
let offset = (point % BLOCK_POINTS) * width;
for (slot, state) in block[offset..offset + width].iter_mut().zip(states) {
if let Some(state) = state {
slot.merge(self.op, FloatState::from_partial(state));
}
.saturating_add(btree_backing_bytes(entries.saturating_add(1))),
)?;
self.retained = self
.retained
.saturating_add(block_bytes)
.saturating_add(btree_backing_bytes(entries.saturating_add(1)))
.saturating_sub(btree_backing_bytes(entries));
entry.insert(vec![FloatState::default(); BLOCK_POINTS * width].into_boxed_slice())
}
};
let offset = (point % BLOCK_POINTS) * width;
for (slot, state) in block[offset..offset + width].iter_mut().zip(states) {
if let Some(state) = state {
slot.merge(self.op, FloatState::from_partial(state));
}
}
Ok(())
@@ -440,6 +606,58 @@ impl FloatMerger {
}
}
/// One present field's state at one position, read from the row's component arrays.
///
/// The components come from the row's own declaration; the native aggregate caller checks
/// compatibility before this runs, so a component list that disagrees is caught before this runs.
fn state_at(
fields: &StateFieldsBatch,
field: usize,
position: usize,
) -> Result<Option<AccumulatorState>> {
if !fields
.is_present(field, position)
.map_err(crate::grid_series::external)?
{
return Ok(None);
}
let mut state = AccumulatorState::default();
for &component in fields.components() {
match component {
StateComponent::IncrementalMean => {
state.incremental_mean = fields
.field_bool_values(field, component)
.map_err(crate::grid_series::external)?
.and_then(|array| (!array.is_null(position)).then(|| array.value(position)))
.ok_or_else(|| {
DataFusionError::Execution(
"present aggregate state has a missing component".into(),
)
})?;
}
_ => {
let value = fields
.field_values(field, component)
.map_err(crate::grid_series::external)?
.and_then(|array| (!array.is_null(position)).then(|| array.value(position)))
.ok_or_else(|| {
DataFusionError::Execution(
"present aggregate state has a missing component".into(),
)
})?;
match component {
StateComponent::Value => state.value = value,
StateComponent::Mean => state.mean = value,
StateComponent::Compensation => state.compensation = value,
StateComponent::Count => state.count = value,
StateComponent::IncrementalMean => unreachable!(),
}
}
}
}
Ok(Some(state))
}
impl FloatState {
fn partial(&self) -> Option<AccumulatorState> {
self.seen.then_some(AccumulatorState {
@@ -575,3 +793,137 @@ pub fn merge_count_values(
}
count_series(schema, groups)
}
#[cfg(test)]
mod tests {
use super::*;
use datafusion::arrow::array::{Float64Array, ListArray, StructArray};
use datafusion::arrow::buffer::{NullBuffer, OffsetBuffer, ScalarBuffer};
use datafusion::arrow::datatypes::{DataType, Field};
use datatypes::series::aggregate_state::{AggregateStateSchema, StateComponent};
use datatypes::series::samples::FieldSpec;
use datatypes::series::time::{EvalGrid, GridRange, GridSpec, TimeAxis};
fn fields_batch() -> StateFieldsBatch {
let item_fields: datafusion::arrow::datatypes::Fields = vec![
Field::new("value", DataType::Float64, true),
Field::new("compensation", DataType::Float64, true),
]
.into();
let make_field =
|values: Vec<Option<f64>>, compensation: Vec<Option<f64>>, validity: Vec<bool>| {
let item = StructArray::new(
item_fields.clone(),
vec![
Arc::new(Float64Array::from(values)),
Arc::new(Float64Array::from(compensation)),
],
Some(NullBuffer::from(validity)),
);
Arc::new(ListArray::new(
Arc::new(Field::new(
"item",
DataType::Struct(item_fields.clone()),
true,
)),
OffsetBuffer::new(ScalarBuffer::from(vec![0i32, 3])),
Arc::new(item),
None,
))
};
let field_a = make_field(
vec![Some(2.), Some(3.), Some(4.)],
vec![Some(0.), Some(0.), Some(0.)],
vec![true, true, true],
);
let field_b = make_field(
vec![Some(5.), None, Some(7.)],
vec![Some(0.), None, Some(0.)],
vec![true, false, true],
);
let outer_fields = vec![
Field::new("a", field_a.data_type().clone(), false),
Field::new("b", field_b.data_type().clone(), false),
]
.into();
let outer = StructArray::new(outer_fields, vec![field_a, field_b], None);
let schema = AggregateStateSchema::try_new(
AggregateStateKind::Sum,
vec![StateComponent::Value, StateComponent::Compensation],
)
.unwrap();
StateFieldsBatch::try_from_arrow(outer, &schema).unwrap()
}
fn schema() -> Arc<SeriesSchema> {
Arc::new(SeriesSchema {
fields: vec![
FieldSpec { name: "a".into() },
FieldSpec { name: "b".into() },
]
.into(),
eval_grid: Some(Arc::new(
EvalGrid::try_new(GridSpec::try_new(0, 1, 3).unwrap()).unwrap(),
)),
})
}
#[test]
fn merge_row_matches_partial_merge_with_field_hole() {
let schema = schema();
let metric = SeriesLabels::try_new(Some("m".into()), Vec::new()).unwrap();
let row = StateRow {
metric: metric.clone(),
time: TimeAxis::Grid(
GridRange::try_new(schema.eval_grid.clone().unwrap(), 0, 3).unwrap(),
),
fields: fields_batch(),
kind: AggregateStateKind::Sum,
};
let mut from_row = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
from_row.merge_row(&row, |_| Ok(())).unwrap();
let mut from_partial = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
from_partial
.merge(PartialAggregate {
op: AggregateOp::Sum,
groups: vec![StateSeries {
metric,
time: TimeAxis::Grid(
GridRange::try_new(schema.eval_grid.clone().unwrap(), 0, 3).unwrap(),
),
states: vec![
vec![
Some(AccumulatorState {
value: 2.0,
..Default::default()
}),
Some(AccumulatorState {
value: 3.0,
..Default::default()
}),
Some(AccumulatorState {
value: 4.0,
..Default::default()
}),
],
vec![
Some(AccumulatorState {
value: 5.0,
..Default::default()
}),
None,
Some(AccumulatorState {
value: 7.0,
..Default::default()
}),
],
],
}],
})
.unwrap();
assert_eq!(
format!("{:?}", from_row.finish().unwrap()),
format!("{:?}", from_partial.finish().unwrap()),
);
}
}
+79 -41
View File
@@ -1394,22 +1394,25 @@ fn assert_flat_close(
fn to_layout(mut partial: PartialAggregate) -> PartialAggregate {
let kept = partial.op.state_fields();
for group in &mut partial.groups {
for (_, states) in &mut group.points {
for state in states.iter_mut().flatten() {
let mut stripped = AccumulatorState::default();
for field in kept {
match field {
StateField::Value => stripped.value = state.value,
StateField::Mean => stripped.mean = state.mean,
StateField::Compensation => stripped.compensation = state.compensation,
StateField::Count => stripped.count = state.count,
StateField::IncrementalMean => {
stripped.incremental_mean = state.incremental_mean
}
for state in group
.states
.iter_mut()
.flat_map(|field| field.iter_mut())
.flatten()
{
let mut stripped = AccumulatorState::default();
for field in kept {
match field {
StateField::Value => stripped.value = state.value,
StateField::Mean => stripped.mean = state.mean,
StateField::Compensation => stripped.compensation = state.compensation,
StateField::Count => stripped.count = state.count,
StateField::IncrementalMean => {
stripped.incremental_mean = state.incremental_mean
}
}
*state = stripped;
}
*state = stripped;
}
}
partial
@@ -1432,7 +1435,7 @@ fn merged_and_serial(op: AggregateOp, partitions: &[&[f64]]) -> (f64, f64) {
aggregator.update(&series).unwrap();
}
merger
.merge(to_layout(aggregator.finish_partial()))
.merge(to_layout(aggregator.finish_partial().unwrap()))
.unwrap();
}
let all: Vec<Series> = partitions
@@ -1471,11 +1474,22 @@ fn finish_partial_reserves_one_slot_per_seen_point() {
aggregator.update(&series).unwrap();
let expected = aggregator
.state_bytes()
.saturating_add(std::mem::size_of::<PartialGroup>())
.saturating_add(std::mem::size_of::<
datatypes::series::aggregate_state::StateSeries,
>())
.saturating_add(std::mem::size_of::<datatypes::series::time::TimeAxis>())
.saturating_add(
SEEN.saturating_mul(std::mem::size_of::<(usize, Vec<Option<AccumulatorState>>)>()),
SEEN.saturating_mul(
std::mem::size_of::<datatypes::series::time::GridSpan>()
.saturating_add(std::mem::size_of::<usize>()),
),
)
.saturating_add(SEEN.saturating_mul(std::mem::size_of::<Option<AccumulatorState>>()));
.saturating_add(std::mem::size_of::<
Vec<Option<datatypes::series::aggregate_state::AggregateStateValues>>,
>())
.saturating_add(SEEN.saturating_mul(std::mem::size_of::<
Option<datatypes::series::aggregate_state::AggregateStateValues>,
>()));
let mut peak = 0usize;
let partial = aggregator
.finish_partial_budgeted(|bytes| {
@@ -1483,9 +1497,11 @@ fn finish_partial_reserves_one_slot_per_seen_point() {
Ok(())
})
.unwrap();
// The vector is reserved once, at the points that carry a state, so pushing does not double it.
assert_eq!(partial.groups[0].points.len(), SEEN);
assert_eq!(partial.groups[0].points.capacity(), SEEN);
// Each field vector is reserved once, at the points that carry a state, so pushing does not
// double it and the export holds one vector per field rather than one per point.
assert_eq!(partial.groups[0].states.len(), 1);
assert_eq!(partial.groups[0].states[0].len(), SEEN);
assert_eq!(partial.groups[0].states[0].capacity(), SEEN);
assert_eq!(peak, expected);
}
@@ -1496,20 +1512,17 @@ fn merge_charges_the_capacity_of_an_input_partial() {
let mut aggregator =
FloatAggregator::try_new(&schema, AggregateOp::Sum, Grouping::By(&[])).unwrap();
aggregator.update(&series).unwrap();
let partial = aggregator.finish_partial();
assert_eq!(partial.groups[0].points.capacity(), 1);
let partial = aggregator.finish_partial().unwrap();
assert_eq!(partial.groups[0].states[0].capacity(), 1);
let mut spare = partial.clone();
let point = spare.groups[0].points.pop().unwrap();
let mut points = Vec::with_capacity(1024);
points.push(point);
spare.groups[0].points = points;
spare.groups[0].states[0].reserve(1023);
let peak = |partial: PartialAggregate| {
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
let (peak, result) = peak_admission(|budget| merger.merge_budgeted(partial, budget));
result.unwrap();
peak
};
let slack = 1023 * std::mem::size_of::<(usize, Vec<Option<AccumulatorState>>)>();
let slack = 1023 * std::mem::size_of::<Option<AccumulatorState>>();
assert!(
peak(spare) >= peak(partial).saturating_add(slack),
"spare capacity was not admitted"
@@ -1526,8 +1539,8 @@ fn merge_charges_one_block_per_new_block_not_per_point() {
let mut aggregator =
FloatAggregator::try_new(&schema, AggregateOp::Sum, Grouping::By(&[])).unwrap();
aggregator.update(&series).unwrap();
let partial = aggregator.finish_partial();
assert_eq!(partial.groups[0].points.len(), 64);
let partial = aggregator.finish_partial().unwrap();
assert_eq!(partial.groups[0].states[0].len(), 64);
let block = BLOCK_POINTS * 64 * std::mem::size_of::<FloatState>();
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
let (peak, result) = peak_admission(|budget| merger.merge_budgeted(partial, budget));
@@ -1545,12 +1558,20 @@ fn merge_charges_one_block_per_new_block_not_per_point() {
#[test]
fn merge_admits_groups_and_blocks_before_it_inserts_them() {
let schema = schema(&["value"]);
let empty_axis = || {
datatypes::series::time::TimeAxis::from_grid_spans(
schema.eval_grid.clone().unwrap(),
Vec::new(),
)
.unwrap()
};
let partial = |ids: std::ops::Range<usize>| PartialAggregate {
op: AggregateOp::Sum,
groups: ids
.map(|id| PartialGroup {
.map(|id| datatypes::series::aggregate_state::StateSeries {
metric: metric("m", &[("i", &id.to_string())]),
points: Vec::new(),
time: empty_axis(),
states: vec![Vec::new()],
})
.collect(),
};
@@ -1564,7 +1585,7 @@ fn merge_admits_groups_and_blocks_before_it_inserts_them() {
assert!(merger.merge_budgeted(partial(100..108), refused).is_err());
assert_eq!(merger.state_bytes(), retained);
let mut invalid = partial(200..201);
invalid.groups[0].points.push((256, Vec::new()));
invalid.groups[0].states[0].push(None);
assert!(merger.merge(invalid).is_err());
assert_eq!(merger.state_bytes(), retained);
assert!(
@@ -1584,7 +1605,7 @@ fn merge_admits_the_key_before_it_builds_it() {
let mut aggregator =
FloatAggregator::try_new(&schema, AggregateOp::Sum, Grouping::By(&[])).unwrap();
aggregator.update(&series).unwrap();
aggregator.finish_partial()
aggregator.finish_partial().unwrap()
};
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
merger.merge(partial()).unwrap();
@@ -1622,9 +1643,14 @@ fn a_long_metric_name_is_held_by_the_group_and_by_its_key() {
let retained = |name: &str| {
let partial = PartialAggregate {
op: AggregateOp::Sum,
groups: vec![PartialGroup {
groups: vec![datatypes::series::aggregate_state::StateSeries {
metric: metric(name, &[]),
points: Vec::new(),
time: datatypes::series::time::TimeAxis::from_grid_spans(
schema.eval_grid.clone().unwrap(),
Vec::new(),
)
.unwrap(),
states: vec![Vec::new()],
}],
};
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
@@ -1747,7 +1773,7 @@ fn partials_merge_to_the_serial_result() {
aggregator.update(series).unwrap();
}
merger
.merge(to_layout(aggregator.finish_partial()))
.merge(to_layout(aggregator.finish_partial().unwrap()))
.unwrap();
}
let merged = merger.finish().unwrap();
@@ -1760,7 +1786,7 @@ fn partials_merge_to_the_serial_result() {
}
let mut merger = FloatMerger::try_new(&schema, op).unwrap();
merger
.merge(to_layout(aggregator.finish_partial()))
.merge(to_layout(aggregator.finish_partial().unwrap()))
.unwrap();
assert_flat_close(
&merger.finish().unwrap(),
@@ -1876,7 +1902,7 @@ fn equal_zero_extremes_follow_the_merge_order() {
aggregator
.update(&input(&schema, metric("m", &[]), 0, &[vec![Some(value)]]))
.unwrap();
merger.merge(aggregator.finish_partial()).unwrap();
merger.merge(aggregator.finish_partial().unwrap()).unwrap();
}
let value = values_of(&merger.finish().unwrap(), 0, 128).1[0].unwrap();
assert_eq!(
@@ -1897,15 +1923,27 @@ fn a_merge_rejects_state_that_does_not_fit() {
aggregator
.update(&input(&schema, metric("m", &[]), 5, &[vec![Some(1.0)]]))
.unwrap();
aggregator.finish_partial()
aggregator.finish_partial().unwrap()
};
let mut merger = FloatMerger::try_new(&schema, AggregateOp::Sum).unwrap();
assert!(merger.merge(partial(AggregateOp::Avg)).is_err());
// A field with a state at a position the axis does not carry: the shape no longer fits.
let mut wide = partial(AggregateOp::Sum);
wide.groups[0].points[0].1.push(None);
wide.groups[0].states[0].push(None);
assert!(merger.merge(wide).is_err());
// An axis whose positions belong to another grid: the merge is bound to its own.
let mut late = partial(AggregateOp::Sum);
late.groups[0].points[0].0 = 128;
// The fixture's grid holds 128 positions, so a position from a longer grid is out of bounds.
let other_grid = Arc::new(EvalGrid::try_new(GridSpec::try_new(0, 1, 256).unwrap()).unwrap());
late.groups[0].time = datatypes::series::time::TimeAxis::from_grid_spans(
other_grid,
vec![datatypes::series::time::GridSpan::new(200, 1)],
)
.unwrap();
late.groups[0].states[0] = vec![Some(AccumulatorState {
value: 1.0,
..Default::default()
})];
assert!(merger.merge(late).is_err());
assert!(merger.finish().unwrap().is_empty());
}
-3
View File
@@ -13,8 +13,6 @@
// limitations under the License.
mod absent;
mod aggregate_exchange;
mod aggregate_finalize;
mod empty_metric;
mod fused_series;
mod histogram_fold;
@@ -31,7 +29,6 @@ mod test_util;
mod union_distinct_on;
pub use absent::{Absent, AbsentExec, AbsentStream};
pub use aggregate_finalize::AggregateFinalize;
use common_query::native_histogram::{SUM_FIELD, native_histogram_value_type};
use common_query::prometheus::is_prometheus_stale_nan;
use datafusion::arrow::array::{Array, Float64Array, StructArray};
@@ -1,486 +0,0 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The private Arrow layout of a partition's partial aggregate state, and the fold that writes it.
//! [`decode_state`] reads a batch back into the rows a `FloatMerger` merges. Only that producer and
//! that merge read [`state_schema`]: it is not a wire format, and has no serde or version.
//!
//! TODO(discord9): Replace this temporary state-row bridge with series frames emitted directly by
//! FusedSeries and consumed by native final aggregation, avoiding the extra state-row encoding,
//! decoding and reconstruction. This private layout is not the final intermediate representation.
#[cfg(test)]
mod tests;
use std::mem::size_of;
use std::sync::{Arc, Mutex};
use datafusion::arrow::array::{
Array, ArrayRef, BooleanArray, Float64Array, StringArray, UInt32Array, UInt64Array,
};
use datafusion::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
use datafusion::arrow::record_batch::RecordBatch;
use datafusion::error::{DataFusionError, Result as DfResult};
use datafusion::execution::memory_pool::{MemoryConsumer, MemoryPool, MemoryReservation};
use datatypes::series::samples::SeriesSchema;
use datatypes::series::{Label, SeriesLabels};
use crate::aggregate::{
AccumulatorState, AggregateOp, FloatAggregator, Grouping, PartialAggregate, PartialGroup,
};
use crate::evaluated::EvaluatedChunk;
use crate::grid_series::series_bytes;
use crate::native::plan::{
AggregateKind, AggregateStage, BoundAggregate, Grouping as BoundGrouping,
};
/// Rows per encoded batch; the exchange has no session context to read a batch size from.
const BATCH_ROWS: usize = 4096;
/// One column's `ArrayRef`, its `Field` and its buffers are all well under this.
const COLUMN_OVERHEAD: usize = 256;
/// The value capacity a string column starts with, nulls or not (Arrow's `StringArray::from`).
const STRING_CAPACITY: usize = 1024;
/// One state row: a group's labels, the point and field it is of, and its state. Label columns are
/// nullable, since absent and empty differ; `__name__` has the metric name's column.
pub(crate) fn state_schema(tags: &[String]) -> SchemaRef {
let mut fields = vec![Field::new("metric_name", DataType::Utf8, true)];
let labels = ordinary_tags(tags).count();
fields.extend((0..labels).map(|at| Field::new(format!("tag_{at}"), DataType::Utf8, true)));
fields.extend([
Field::new("point", DataType::UInt64, false),
Field::new("field", DataType::UInt32, false),
]);
for name in ["value", "mean", "compensation", "count"] {
fields.push(Field::new(name, DataType::Float64, false));
}
fields.push(Field::new("incremental_mean", DataType::Boolean, false));
Arc::new(Schema::new(fields))
}
/// The kernel a `Complete` aggregate binds; `quantile` and the other stages are not a fold.
pub(crate) fn aggregate_op(op: &BoundAggregate) -> DfResult<AggregateOp> {
if op.stage != AggregateStage::Complete {
return Err(DataFusionError::NotImplemented(format!(
"the aggregate state exchange folds only `Complete` aggregates, not `{:?}`",
op.stage
)));
}
match op.kind {
AggregateKind::Sum => Ok(AggregateOp::Sum),
AggregateKind::Count => Ok(AggregateOp::Count),
AggregateKind::Avg => Ok(AggregateOp::Avg),
AggregateKind::Min => Ok(AggregateOp::Min),
AggregateKind::Max => Ok(AggregateOp::Max),
AggregateKind::Group => Ok(AggregateOp::Group),
AggregateKind::Stddev => Ok(AggregateOp::Stddev),
AggregateKind::Stdvar => Ok(AggregateOp::Stdvar),
AggregateKind::Quantile { quantile } => Err(DataFusionError::NotImplemented(format!(
"the aggregate state exchange does not carry `quantile {quantile}`"
))),
}
}
/// One partition's fold, and the state it encodes when its input ends.
pub(crate) struct PartialFold {
tags: Arc<[String]>,
pool: Arc<dyn MemoryPool>,
/// What this fold holds beside the kernel's state, so each charge can be added to it.
held: usize,
reservation: MemoryReservation,
/// The plan's charge for exported batches, which live until its queue drains.
export_budget: Arc<Mutex<Option<MemoryReservation>>>,
fold: Option<FloatAggregator>,
}
impl PartialFold {
/// Starts one partition's fold, its `export_budget` registered against `pool` at first export.
pub(crate) fn try_new(
op: BoundAggregate,
schema: Arc<SeriesSchema>,
tags: Arc<[String]>,
export_budget: Arc<Mutex<Option<MemoryReservation>>>,
pool: &Arc<dyn MemoryPool>,
) -> DfResult<Self> {
let kind = aggregate_op(&op)?;
let per_tag = size_of::<String>() + size_of::<Field>() + size_of::<ArrayRef>();
let held = tags.iter().map(|tag| tag.len() + per_tag).sum::<usize>() + 8 * per_tag;
let reservation = MemoryConsumer::new("AggregateExchangeState").register(pool);
reservation.try_resize(held)?;
// `__name__` has the metric name's column; a list without it is reused, not copied.
let tags: Arc<[String]> = if tags.iter().any(|tag| tag == "__name__") {
ordinary_tags(&tags)
.map(ToOwned::to_owned)
.collect::<Vec<_>>()
.into()
} else {
tags
};
// The grouping names are a scratch copy of the binding's list, which can be longer than the
// tag list, so they are admitted while the kernel is built and given back after.
let scratch = match &op.group {
BoundGrouping::By(names) | BoundGrouping::Without(names) => {
names.len() * size_of::<&str>()
}
};
reservation.try_resize(held.saturating_add(scratch))?;
let fold = {
let names: Vec<&str> = match &op.group {
BoundGrouping::By(names) | BoundGrouping::Without(names) => {
names.iter().map(AsRef::as_ref).collect()
}
};
let grouping = match &op.group {
BoundGrouping::By(_) => Grouping::By(&names),
BoundGrouping::Without(_) => Grouping::Without(&names),
};
FloatAggregator::try_new_budgeted(&schema, kind, grouping, |total| {
reservation.try_resize(held.saturating_add(scratch).saturating_add(total))
})?
};
reservation.try_resize(held)?;
Ok(Self {
tags,
pool: Arc::clone(pool),
held,
reservation,
export_budget,
fold: Some(fold),
})
}
/// Folds one chunk, held while it is folded and dropped after. The kernel validates the chunk
/// against the schema this fold was built with before it charges anything.
pub(crate) fn update(&mut self, chunk: EvaluatedChunk) -> DfResult<()> {
let fold = self.fold.as_mut().ok_or_else(finished)?;
let pending = self.held + series_bytes(&chunk.series);
fold.update_budgeted(&chunk.series, |fold_bytes| {
self.reservation.try_resize(pending + fold_bytes)
})?;
drop(chunk);
self.reservation.try_resize(self.held + fold.state_bytes())
}
/// Encodes the fold's state once the caller has read the input to its end. The batches stay
/// charged to [`Self::export_budget`] after the states are given back, since they outlive them.
pub(crate) fn finish(&mut self) -> DfResult<Vec<RecordBatch>> {
let fold = self.fold.take().ok_or_else(finished)?;
let partial =
fold.finish_partial_budgeted(|total| self.reservation.try_resize(self.held + total))?;
let batches = self.encode(&partial)?;
drop(partial);
self.reservation.try_resize(self.held)?;
Ok(batches)
}
/// Encodes `partial` as one row buffer, admitted in full before it allocates, and hands it over
/// as slices of at most [`BATCH_ROWS`] rows that share it.
fn encode(&self, partial: &PartialAggregate) -> DfResult<Vec<RecordBatch>> {
let mut rows = 0usize;
let mut label_bytes = 0usize;
for group in &partial.groups {
let labels = group.metric.name().map_or(0, str::len).saturating_add(
ordinary_tags(&self.tags)
.filter_map(|tag| group.metric.get(tag).map(|value| tag.len() + value.len()))
.sum::<usize>(),
);
let present: usize = group
.points
.iter()
.map(|p| p.1.iter().flatten().count())
.sum();
rows += present;
label_bytes += labels * present;
}
if rows > 0 {
self.grow_export(export_bytes(rows, label_bytes, self.tags.len()))?;
}
let mut metric_name: Vec<Option<&str>> = Vec::with_capacity(rows);
let mut tags: Vec<Vec<_>> = self.tags.iter().map(|_| Vec::with_capacity(rows)).collect();
let (mut point, mut field) = (Vec::with_capacity(rows), Vec::with_capacity(rows));
let mut components: [Vec<f64>; 4] = std::array::from_fn(|_| Vec::with_capacity(rows));
let mut incremental_mean: Vec<bool> = Vec::with_capacity(rows);
for group in &partial.groups {
// A group's labels are the same for every row it writes, so resolve each tag once
// instead of once per row. `SeriesLabels::get` binary-searches the group's tags, and
// this loop runs once per (point, field) of the group.
let resolved: Vec<Option<&str>> =
self.tags.iter().map(|tag| group.metric.get(tag)).collect();
for (at, states) in &group.points {
for (index, state) in states.iter().enumerate() {
let Some(state) = state else {
continue;
};
metric_name.push(group.metric.name());
for (column, value) in tags.iter_mut().zip(resolved.iter()) {
column.push(*value);
}
point.push(*at as u64);
field.push(index as u32);
components[0].push(state.value);
components[1].push(state.mean);
components[2].push(state.compensation);
components[3].push(state.count);
incremental_mean.push(state.incremental_mean);
}
}
}
let mut columns: Vec<ArrayRef> = Vec::with_capacity(8 + tags.len());
columns.push(Arc::new(StringArray::from(metric_name)));
for values in tags {
columns.push(Arc::new(StringArray::from(values)));
}
columns.push(Arc::new(UInt64Array::from(point)));
columns.push(Arc::new(UInt32Array::from(field)));
for values in components {
columns.push(Arc::new(Float64Array::from(values)));
}
columns.push(Arc::new(BooleanArray::from(incremental_mean)));
let batch = RecordBatch::try_new(state_schema(&self.tags), columns)?;
Ok((0..rows)
.step_by(BATCH_ROWS)
.map(|offset| batch.slice(offset, BATCH_ROWS.min(rows - offset)))
.collect())
}
/// Admits `bytes` to the plan's export reservation, registered against this pool at first use.
fn grow_export(&self, bytes: usize) -> DfResult<()> {
let mut budget = self.export_budget.lock().unwrap_or_else(|e| e.into_inner());
let consumer = MemoryConsumer::new("AggregateExchangeExport");
budget
.get_or_insert_with(|| consumer.register(&self.pool))
.try_grow(bytes)
}
}
/// Reads one state batch back into the rows a `FloatMerger` merges.
///
/// The encoder writes a group's points in ascending order, one row per (point, field), so a label
/// change opens the next group and everything else extends the current one: consecutive rows of a
/// point fill that point's state vector, and a later point is appended to the same group. Rows of
/// one group split across batches, or concatenated back to back, start a further point entry
/// instead of failing; the merge addresses points by index, so both entries fold into one slot.
pub(crate) fn decode_state(
batch: &RecordBatch,
schema: &Arc<SeriesSchema>,
op: AggregateOp,
tags: &[String],
mut budget: impl FnMut(usize) -> DfResult<()>,
) -> DfResult<PartialAggregate> {
// The expected layout builds a name and a field per column, so it is admitted before it is
// built, together with the column lookups below.
let mut total = (tags.len() + 9)
.saturating_mul(size_of::<&str>() + size_of::<String>() + size_of::<Field>());
budget(total)?;
let layout = state_schema(tags);
let batch_schema = batch.schema_ref();
if batch_schema.as_ref() != layout.as_ref() {
return Err(DataFusionError::Execution(format!(
"not the layout {layout:?}"
)));
}
let grid = schema.eval_grid.as_ref().ok_or_else(|| {
DataFusionError::Execution("aggregate state is indexed on an evaluated grid".into())
})?;
let width = schema.fields.len();
// The name list and the label column lookups are workspace, sized exactly and charged before
// either allocates.
total = total.saturating_add(tags.len() * (size_of::<&str>() + size_of::<&StringArray>()));
budget(total)?;
let mut names: Vec<&str> = Vec::with_capacity(tags.len());
names.extend(ordinary_tags(tags));
let metric_name: &StringArray = column(batch, "metric_name")?;
let mut tag_columns: Vec<&StringArray> = Vec::with_capacity(names.len());
for at in 0..names.len() {
tag_columns.push(column(batch, &format!("tag_{at}"))?);
}
let point: &UInt64Array = column(batch, "point")?;
let field: &UInt32Array = column(batch, "field")?;
let value: &Float64Array = column(batch, "value")?;
let mean: &Float64Array = column(batch, "mean")?;
let compensation: &Float64Array = column(batch, "compensation")?;
let count: &Float64Array = column(batch, "count")?;
let incremental_mean: &BooleanArray = column(batch, "incremental_mean")?;
// A null in a state column is not a zero: that state was never folded. Only the label columns
// ahead of these are nullable.
for at in names.len() + 1..batch.num_columns() {
let nulls = batch.column(at).null_count();
let name = batch_schema.field(at).name();
if nulls > 0 {
return Err(DataFusionError::Execution(format!(
"the aggregate state column `{name}` holds {nulls} nulls, a state being a value"
)));
}
}
let mut groups: Vec<PartialGroup> = Vec::new();
for row in 0..batch.num_rows() {
let at = usize::try_from(point.value(row)).unwrap_or(usize::MAX);
let field = field.value(row) as usize;
if at >= grid.spec().len as usize || field >= width {
return Err(DataFusionError::Execution(format!(
"aggregate state row {row} is of point {at} of the grid's {} or of field {field} \
of the aggregate's {width}",
grid.spec().len
)));
}
// The encoder writes a group's points in ascending order and its fields together, so only a
// label change opens a group: a row repeating the labels of the row before it is the same
// group, and is that point's next field or the group's next point. Comparing a label change
// against a change of point would open one group per row, since consecutive points of one
// group differ by definition.
let differs = |column: &StringArray| {
column.is_null(row - 1) != column.is_null(row)
|| (!column.is_null(row) && column.value(row - 1) != column.value(row))
};
let opens =
row == 0 || differs(metric_name) || tag_columns.iter().any(|column| differs(column));
if opens {
// A push outgrowing the group vector doubles it, so the block it is about to allocate
// is admitted here; the point's tuple and state vector and the label scratch follow
// before either allocates, then what the labels carry once they are read.
let growth = if groups.len() == groups.capacity() {
groups.capacity().max(1) * 2 * size_of::<PartialGroup>()
} else {
0
};
total = total.saturating_add(
growth
+ size_of::<(usize, Vec<Option<AccumulatorState>>)>()
+ width * size_of::<Option<AccumulatorState>>()
+ (names.len() + 1) * size_of::<(&str, &str)>(),
);
budget(total)?;
let name = (!metric_name.is_null(row)).then(|| metric_name.value(row));
let mut carried: Vec<(&str, &str)> = Vec::with_capacity(names.len());
for (column, label) in tag_columns.iter().zip(&names) {
if !column.is_null(row) {
carried.push((*label, column.value(row)));
}
}
// The label vector, the `Arc<str>` header each name and value points at, and the bytes
// they hold. One extra pair is the metric name.
total = total.saturating_add(
(names.len() + 1) * (size_of::<Label>() + size_of::<usize>())
+ name.map_or(0, str::len)
+ carried
.iter()
.map(|(label, value)| label.len() + value.len())
.sum::<usize>(),
);
budget(total)?;
let mut labels: Vec<Label> = Vec::with_capacity(carried.len());
for (label, value) in carried {
labels.push(Label {
name: Arc::from(label),
value: Arc::from(value),
});
}
let metric = SeriesLabels::try_new(name.map(Arc::from), labels).map_err(|error| {
DataFusionError::Execution(format!(
"aggregate state row {row} does not carry a label set: {error}"
))
})?;
groups.push(PartialGroup {
metric,
points: vec![(at, vec![None; width])],
});
} else {
// The same group as the row before: this row's point is the one its state belongs to.
// The encoder emits a group's points in ascending order, so a later point is appended
// to the group rather than starting a new one.
let group = groups.last_mut().ok_or_else(|| {
DataFusionError::Execution(format!("aggregate state row {row} continued no group"))
})?;
// A batch that holds two encoders' output back to back restarts a group at an earlier
// point, and repeats a (point, field) pair. Both are accepted: the merge addresses
// points by index, so a second entry for the same point folds into the same slot,
// exactly as merging the two batches separately would.
let opens_point = match group.points.last() {
Some((last, states)) => *last != at || states[field].is_some(),
None => true,
};
if opens_point {
// The point vector doubles when it outgrows itself, then the point's tuple and its
// state vector follow, both admitted before either allocates.
let growth = if group.points.len() == group.points.capacity() {
group.points.capacity().max(1)
* 2
* size_of::<(usize, Vec<Option<AccumulatorState>>)>()
} else {
0
};
total = total.saturating_add(
growth
+ size_of::<(usize, Vec<Option<AccumulatorState>>)>()
+ width * size_of::<Option<AccumulatorState>>(),
);
budget(total)?;
group.points.push((at, vec![None; width]));
}
}
let group = groups.last_mut().ok_or_else(|| {
DataFusionError::Execution(format!("aggregate state row {row} opened no group"))
})?;
let (_, states) = group.points.last_mut().ok_or_else(|| {
DataFusionError::Execution(format!("aggregate state row {row} continued no point"))
})?;
states[field] = Some(AccumulatorState {
value: value.value(row),
mean: mean.value(row),
compensation: compensation.value(row),
count: count.value(row),
incremental_mean: incremental_mean.value(row),
});
}
Ok(PartialAggregate { op, groups })
}
/// The caller's tag names in column order, without the metric name, which has its own column.
fn ordinary_tags(tags: &[String]) -> impl Iterator<Item = &str> + '_ {
tags.iter()
.filter(|tag| tag.as_str() != "__name__")
.map(String::as_str)
}
/// The error for a fold that has already been finished.
fn finished() -> DataFusionError {
DataFusionError::Execution("the partial aggregate already finished its fold".into())
}
/// An upper bound on one export: the scratch row vectors, the Arrow arrays they are frozen into --
/// which live beside them while the strings are copied -- and the slices the output holds.
///
/// A string column starts at [`STRING_CAPACITY`] value bytes even when every row is null, and the
/// builder copies what it appended before handing its buffer over, so one minimum capacity is
/// admitted per column beside the label payload twice over. Coarse on purpose: an overestimate.
fn export_bytes(rows: usize, label_bytes: usize, tags: usize) -> usize {
let strings = tags.saturating_add(1);
let row = size_of::<u64>() + size_of::<u32>() + 4 * size_of::<f64>() + size_of::<bool>();
rows * row
+ strings * rows * size_of::<Option<&str>>()
+ strings * (rows + 1) * size_of::<i32>()
+ strings * (rows / 8 + 1)
+ strings * STRING_CAPACITY
+ 2 * label_bytes
+ (strings + 8) * COLUMN_OVERHEAD
+ rows.div_ceil(BATCH_ROWS) * size_of::<RecordBatch>()
}
/// Reads the column `name` as `A`, which the layout check has already made certain.
fn column<'a, A: 'static>(batch: &'a RecordBatch, name: &str) -> DfResult<&'a A> {
let found = batch
.column_by_name(name)
.and_then(|c| c.as_any().downcast_ref::<A>());
found.ok_or_else(|| DataFusionError::Execution(format!("the column `{name}` has another type")))
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -1,745 +0,0 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! What the split has to keep right: which aggregate is cut out, what stays above the merge, and
//! which shapes are declined so the caller can keep the plan it had.
//!
//! The trees are built node by node with the same bindings the PromQL compiler produces, because the
//! compiler lives in the `query` crate, which depends on this one.
use super::*;
use datafusion::arrow::datatypes::{Field, TimeUnit};
use datafusion::common::DFSchema;
use datatypes::series::samples::FieldSpec;
use datatypes::series::time::{EvalGrid, GridSpec};
use crate::histogram_eval::HistogramFunction;
use crate::native::plan::BoundVectorOp;
use crate::select::SortOrder;
use crate::vector_match::{BinaryMatching, BinaryOp, Cardinality, FillValues, MatchingLabels};
/// The statement's grid, which every evaluated chunk is addressed on.
fn grid() -> Arc<EvalGrid> {
Arc::new(EvalGrid::try_new(GridSpec::try_new(0, 60_000, 4).unwrap()).unwrap())
}
/// The raw fields the tree's input binds, which a scan below it has to be able to produce.
fn raw_fields() -> Arc<[FieldSpec]> {
Arc::from([FieldSpec { name: "v0".into() }])
}
/// The raw samples a scan hands over: one field, no grid.
fn raw() -> NativeNode {
let output = NativeType::Samples(Arc::new(SeriesSchema {
fields: raw_fields(),
eval_grid: None,
}));
NativeNode {
expr: NativeExpr::Input(BoundInput {
source_index: 0,
output: output.clone(),
ordering: SeriesOrdering::KeyThenTime,
}),
output,
}
}
/// The evaluated layout a selection or a window reports.
fn evaluated() -> NativeType {
NativeType::Samples(Arc::new(SeriesSchema {
fields: Arc::from([FieldSpec { name: "v0".into() }]),
eval_grid: Some(grid()),
}))
}
/// `rate(v0[5m])`, the per-series prefix of every aggregate below.
fn window() -> NativeNode {
NativeNode {
expr: NativeExpr::Window {
input: Box::new(raw()),
op: crate::native::plan::BoundWindow {
function: crate::range_eval::RangeFunction::Rate,
window: crate::range_eval::RangeWindow {
range_ms: 300_000,
offset_ms: 0,
anchor_ms: None,
modifier: None,
},
input_grid: None,
output_grid: grid(),
},
},
output: evaluated(),
}
}
/// An instant selection, the other per-series prefix the split admits.
fn instant() -> NativeNode {
NativeNode {
expr: NativeExpr::Instant {
input: Box::new(raw()),
op: crate::native::plan::BoundInstant {
lookback_ms: 300_000,
offset_ms: 0,
at_ms: None,
value: crate::native::plan::InstantValue::Sample,
output_grid: grid(),
},
},
output: evaluated(),
}
}
fn map_over(inner: NativeNode, ops: Vec<MathOp>) -> NativeNode {
let output = inner.output.clone();
NativeNode {
expr: NativeExpr::Map {
input: Box::new(inner),
op: crate::native::plan::BoundMap {
ops: ops.into_iter().map(MapOp::Math).collect::<Vec<_>>().into(),
},
},
output,
}
}
fn aggregate_over(
inner: NativeNode,
kind: AggregateKind,
group: Grouping,
stage: AggregateStage,
) -> NativeNode {
let output = inner.output.clone();
NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(inner),
op: BoundAggregate { kind, group, stage },
},
output,
}
}
fn complete(kind: AggregateKind, group: Grouping) -> impl Fn(NativeNode) -> NativeNode {
move |inner| aggregate_over(inner, kind, group.clone(), AggregateStage::Complete)
}
fn by(labels: &[&str]) -> Grouping {
Grouping::By(
labels
.iter()
.map(|label| Arc::<str>::from(*label))
.collect(),
)
}
/// The plan below the producer: a scan producing the raw fields the tree's input binds.
///
/// The producer checks that its child can produce every field its tree input declares, so the scan
/// names them as nullable float columns; the merge itself reads none of them.
fn scan() -> LogicalPlan {
let columns = raw_fields()
.iter()
.map(|field| {
(
None,
Arc::new(Field::new(field.name.as_ref(), DataType::Float64, true)),
)
})
.collect::<Vec<_>>();
LogicalPlan::EmptyRelation(datafusion::logical_expr::EmptyRelation {
produce_one_row: false,
schema: Arc::new(DFSchema::new_with_metadata(columns, Default::default()).unwrap()),
})
}
fn binding() -> FusedSeriesBinding {
FusedSeriesBinding {
grid: grid(),
metric_name: Some(Arc::from("m")),
time_index: "ts".to_string(),
tags: vec!["k0".to_string(), "k1".to_string()],
}
}
/// The rows a finalized aggregate reports: the time index, one value column and the grouped labels.
fn output_schema(values: &[&str], tags: &[&str]) -> DFSchemaRef {
let mut columns = vec![(
None,
Arc::new(Field::new(
"ts",
DataType::Timestamp(TimeUnit::Millisecond, None),
false,
)),
)];
for name in values {
columns.push((None, Arc::new(Field::new(*name, DataType::Float64, true))));
}
for name in tags {
columns.push((None, Arc::new(Field::new(*name, DataType::Utf8, true))));
}
Arc::new(DFSchema::new_with_metadata(columns, Default::default()).unwrap())
}
fn finalize(root: NativeNode, output: DFSchemaRef) -> DfResult<AggregateFinalize> {
AggregateFinalize::try_new(
Arc::from("sum(rate(m[5m]))"),
NativeTree::try_new(root)?,
binding(),
scan(),
output,
)
}
/// The labels this node writes, read off its schema the way the physical half reads them.
fn output_tags(node: &AggregateFinalize) -> Vec<String> {
node.schema()
.fields()
.iter()
.filter(|field| field.data_type() == &DataType::Utf8)
.map(|field| field.name().clone())
.collect()
}
/// The producer this node merges.
fn producer(node: &AggregateFinalize) -> &FusedSeries {
let LogicalPlan::Extension(Extension { node }) = &node.input else {
panic!("the merge should read a producer");
};
node.as_any()
.downcast_ref::<FusedSeries>()
.expect("the producer is a fused series node")
}
/// The innermost aggregate is the one cut out, and only the nodes that may sit above it keep their
/// place: the outer aggregate runs in the suffix, over the merged values.
#[test]
fn the_innermost_aggregate_is_cut_out_of_a_nested_tree() {
let inner = complete(AggregateKind::Avg, by(&["k0"]))(window());
let root = complete(AggregateKind::Sum, by(&[]))(inner);
let node = finalize(root, output_schema(&["sum(avg(v0))"], &[])).unwrap();
// The producer runs the per-series prefix and exports the inner aggregate's state.
let producer = producer(&node);
assert_eq!(
Some(AggregateKind::Avg),
producer.partial_fold().map(|op| op.kind)
);
let prefix = producer.tree().to_string();
assert!(prefix.contains("Window"), "{prefix}");
assert!(!prefix.contains("Aggregate"), "{prefix}");
// The suffix starts at the merged values and holds the outer aggregate.
let suffix = node.suffix.to_string();
assert!(suffix.contains("Aggregate"), "{suffix}");
assert!(suffix.contains("Input"), "{suffix}");
assert!(!suffix.contains("Window"), "{suffix}");
}
/// A map above the merge is point-wise, so it is the same operation over merged values.
#[test]
fn an_outer_map_stays_in_the_suffix() {
let root = map_over(
complete(AggregateKind::Sum, by(&["k0"]))(instant()),
vec![MathOp::Ceil],
);
let node = finalize(root, output_schema(&["ceil(sum(v0))"], &["k0"])).unwrap();
let suffix = node.suffix.to_string();
assert!(suffix.contains("Map"), "{suffix}");
assert!(suffix.contains("Input"), "{suffix}");
assert_eq!(output_tags(&node), ["k0".to_string()]);
}
/// `left <op> right` with a constant on one side, bound the way the compiler binds a broadcast: the
/// operation reports the vector side's samples.
fn broadcast(left: NativeNode, right: NativeNode, op: BinaryOp) -> NativeNode {
let output = if matches!(left.output, NativeType::Scalar(_)) {
right.output.clone()
} else {
left.output.clone()
};
NativeNode {
expr: NativeExpr::Binary {
left: Box::new(left),
right: Box::new(right),
matching: BinaryMatching {
op,
return_bool: false,
labels: MatchingLabels::Ignoring(Vec::new()),
cardinality: Cardinality::OneToOne,
fill: FillValues::default(),
},
},
output,
}
}
/// The constant scalar `value`, on the statement's grid.
fn scalar_on(value: f64) -> NativeNode {
NativeNode {
expr: NativeExpr::Scalar(crate::native::plan::BoundScalar { value }),
output: NativeType::Scalar(grid()),
}
}
/// A constant operand is broadcast over the merged values, so the aggregate under the other side is
/// still the one cut: the suffix keeps the operation, the operand order and the matching, with the
/// merged values where the aggregate was.
#[test]
fn a_constant_operand_keeps_the_cut_below_it() {
for (constant_on_the_left, value) in [(true, "100 - avg(v0)"), (false, "avg(v0) - 100")] {
let aggregate = complete(AggregateKind::Avg, by(&["k0"]))(window());
let root = if constant_on_the_left {
broadcast(scalar_on(100.0), aggregate, BinaryOp::Sub)
} else {
broadcast(aggregate, scalar_on(100.0), BinaryOp::Sub)
};
let node = finalize(root, output_schema(&[value], &["k0"])).unwrap();
// The producer still runs only the per-series prefix: the constant is above the cut, so the
// window is all the prefix holds and the constant is applied in the suffix.
let prefix = producer(&node).tree().to_string();
assert!(prefix.contains("Window"), "{prefix}");
assert!(!prefix.contains("Binary"), "{prefix}");
assert!(!prefix.contains("Scalar"), "{prefix}");
// The suffix is the operation over the merged values, which enter it as its own source `0`:
// the prefix's and the suffix's inputs are separate executions, not one exchange.
let NativeExpr::Binary {
left,
right,
matching,
} = &node.suffix.root().expr
else {
panic!("the operation stays above the cut: {}", node.suffix);
};
assert_eq!(BinaryOp::Sub, matching.op, "the operator is kept");
let (constant, merged) = if constant_on_the_left {
(left, right)
} else {
(right, left)
};
assert!(
matches!(constant.expr, NativeExpr::Scalar(_)),
"the constant keeps its side: {}",
node.suffix
);
assert!(
matches!(&merged.expr, NativeExpr::Input(input) if input.source_index == 0),
"the merged values enter where the aggregate was: {}",
node.suffix
);
assert!(!node.suffix.to_string().contains("Window"));
}
}
fn vector_over(input: NativeNode, op: BoundVectorOp) -> NativeNode {
let output = input.output.clone();
NativeNode {
expr: NativeExpr::Vector {
input: Box::new(input),
op,
},
output,
}
}
#[test]
fn topk_and_bottomk_run_after_the_aggregate_cut() {
for top in [true, false] {
let aggregate = complete(AggregateKind::Sum, by(&["k0"]))(window());
let root = vector_over(
aggregate,
BoundVectorOp::TopK {
k: 10.0,
top,
group: by(&[]),
},
);
let tree = NativeTree::try_new(root).unwrap();
assert!(AggregateFinalize::can_split(&tree, &raw_fields(), "ts"));
let node = finalize(tree.root().clone(), output_schema(&["sum(v0)"], &["k0"])).unwrap();
let NativeExpr::Vector {
input,
op: BoundVectorOp::TopK {
top: actual_top, ..
},
} = &node.suffix.root().expr
else {
panic!("{}", node.suffix);
};
assert_eq!(top, *actual_top);
assert!(
matches!(&input.expr, NativeExpr::Input(bound) if bound.source_index == 0 && bound.output.samples().is_some_and(|schema| schema.fields.len() == 1 && schema.fields[0].name.as_ref() == "v0" && schema.eval_grid.as_ref().is_some_and(|actual| actual.spec() == grid().spec())))
);
}
}
/// A classic histogram function above the cut reads each histogram off the bucket series that share
/// its other labels, and the merge has already folded every partition's `(..., le)` group into one
/// complete series: the set it reads is the one the statement's input would have produced in one
/// execution, so this is the same computation and not an approximation. This is the o11ybench Q10
/// shape.
#[test]
fn histogram_quantile_over_an_aggregate_splits() {
let tree = NativeTree::try_new(vector_over(
complete(AggregateKind::Sum, by(&["instance", "le"]))(window()),
BoundVectorOp::Histogram(HistogramFunction::Quantile(0.99)),
))
.unwrap();
assert!(AggregateFinalize::can_split(&tree, &raw_fields(), "ts"));
// The function stays above the cut, reading the merged series at its own source `0`.
let node = finalize(
tree.root().clone(),
output_schema(&["sum(v0)"], &["instance", "le"]),
)
.unwrap();
let NativeExpr::Vector {
input,
op: BoundVectorOp::Histogram(HistogramFunction::Quantile(quantile)),
} = &node.suffix.root().expr
else {
panic!(
"the histogram function stays above the cut: {}",
node.suffix
);
};
assert_eq!(0.99, *quantile);
assert!(
matches!(&input.expr, NativeExpr::Input(bound) if bound.source_index == 0),
"the merged values enter where the aggregate was: {}",
node.suffix
);
// The prefix is the per-series tree: the histogram function is not one partition's to compute.
let prefix = producer(&node).tree().to_string();
assert!(!prefix.contains("histogram_quantile"), "{prefix}");
assert!(!prefix.contains("Aggregate"), "{prefix}");
assert!(prefix.contains("Window"), "{prefix}");
}
/// Admitting the histogram functions widens one arm, not the whole `Vector` family: every other
/// vector operation is still declined, `sort` here because it reorders the input rather than reading
/// each series of it.
#[test]
fn a_vector_operation_that_is_neither_topk_nor_a_histogram_is_declined() {
let tree = NativeTree::try_new(vector_over(
complete(AggregateKind::Sum, by(&["instance"]))(window()),
BoundVectorOp::Sort(SortOrder::Asc),
))
.unwrap();
assert!(!AggregateFinalize::can_split(&tree, &raw_fields(), "ts"));
let error = finalize(
tree.root().clone(),
output_schema(&["sum(v0)"], &["instance"]),
)
.unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
#[test]
fn limitk_above_the_cut_is_declined_because_merge_does_not_preserve_input_order() {
let error = finalize(
vector_over(
complete(AggregateKind::Sum, by(&["k0"]))(window()),
BoundVectorOp::LimitK {
k: 1.0,
group: by(&[]),
},
),
output_schema(&["sum(v0)"], &["k0"]),
)
.unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
#[test]
fn topk_without_aggregate_is_not_split() {
let tree = NativeTree::try_new(vector_over(
window(),
BoundVectorOp::TopK {
k: 10.0,
top: true,
group: by(&[]),
},
))
.unwrap();
assert!(!AggregateFinalize::can_split(&tree, &raw_fields(), "ts"));
assert!(split(tree.root(), &raw_fields(), "ts").unwrap().is_none());
}
/// A binary operation above the cut is only the operation over merged values when its other side is a
/// constant: two aggregates are two cuts, a vector sibling would have to be matched with the merged
/// values by label, and a prefix that pairs two vectors reads the whole vector rather than one series.
#[test]
fn a_binary_operation_without_a_constant_operand_is_refused() {
let folded = || {
complete(AggregateKind::Sum, by(&[]))(complete(AggregateKind::Avg, by(&["k0"]))(window()))
};
for root in [
// Two sides fold, and one cut cannot answer two aggregates.
broadcast(folded(), folded(), BinaryOp::Add),
// A vector sibling is another series set, which nothing below the merge brings together.
broadcast(
complete(AggregateKind::Sum, by(&["k0"]))(window()),
window(),
BinaryOp::Add,
),
broadcast(
window(),
complete(AggregateKind::Sum, by(&["k0"]))(window()),
BinaryOp::Div,
),
// A prefix that pairs two vectors reads the whole vector at a point, which one partition
// cannot do: admitting a constant operand does not admit this.
complete(AggregateKind::Sum, by(&["k0"]))(broadcast(window(), window(), BinaryOp::Div)),
] {
let error = finalize(root, output_schema(&["sum(v0)"], &["k0"])).unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
}
/// The output's labels are the schema's string columns, which is what the merge writes.
#[test]
fn the_output_labels_are_the_schema_labels() {
let root = complete(AggregateKind::Sum, by(&["k0", "k1"]))(window());
let node = finalize(root, output_schema(&["sum(v0)"], &["k0", "k1"])).unwrap();
assert_eq!(output_tags(&node), ["k0".to_string(), "k1".to_string()]);
}
/// An aggregate directly over the raw input never reaches the split: the tree's own validation
/// refuses a `Complete` aggregate that consumes raw samples, so nothing here has a prefix to run.
#[test]
fn a_raw_aggregate_is_refused_at_the_tree_boundary() {
let error = NativeTree::try_new(complete(AggregateKind::Sum, by(&["k0"]))(raw())).unwrap_err();
assert!(matches!(error, DataFusionError::Internal(_)), "{error}");
assert!(
error.to_string().contains("evaluated vector"),
"the refusal names the input a complete aggregate needs: {error}"
);
}
/// A window over an evaluated input is a subquery, evaluated on its own grid, so what the merge
/// produces is not its input: the split keeps the existing plan.
#[test]
fn a_subquery_above_an_aggregate_is_refused() {
let root = complete(AggregateKind::Sum, by(&["k0"]))(NativeNode {
expr: NativeExpr::Window {
input: Box::new(complete(AggregateKind::Sum, by(&["k0"]))(window())),
op: crate::native::plan::BoundWindow {
function: crate::range_eval::RangeFunction::Rate,
window: crate::range_eval::RangeWindow {
range_ms: 300_000,
offset_ms: 0,
anchor_ms: None,
modifier: None,
},
input_grid: Some(grid()),
output_grid: grid(),
},
},
output: evaluated(),
});
let error = finalize(root, output_schema(&["sum(v0)"], &["k0"])).unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
/// `clamp` with its maximum below its minimum answers nothing, so the fold would see a group with no
/// sample rather than the query the reader wrote.
#[test]
fn an_empty_clamp_is_refused() {
let empty = map_over(instant(), vec![MathOp::Clamp { min: 2.0, max: 1.0 }]);
let error = finalize(
complete(AggregateKind::Sum, by(&["k0"]))(empty),
output_schema(&["sum(v0)"], &["k0"]),
)
.unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
// The same clamp the other way round is a range, and the prefix keeps it.
let range = map_over(instant(), vec![MathOp::Clamp { min: 0.0, max: 1.0 }]);
assert!(
finalize(
complete(AggregateKind::Sum, by(&["k0"]))(range),
output_schema(&["sum(v0)"], &["k0"])
)
.is_ok()
);
}
/// A group that varies with a value field or the grid point is not a group the state rows can name.
#[test]
fn an_aggregate_grouped_by_a_value_field_or_the_time_index_is_refused() {
for group in [by(&["v0"]), by(&["ts"])] {
let error = finalize(
complete(AggregateKind::Sum, group)(window()),
output_schema(&["sum(v0)"], &["v0"]),
)
.unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
// A label the input does not carry is simply absent, as in the existing planner.
assert!(
finalize(
complete(AggregateKind::Sum, by(&["absent"]))(window()),
output_schema(&["sum(v0)"], &["absent"])
)
.is_ok()
);
}
/// A quantile's state is the group's samples, which neither half of this node can carry.
#[test]
fn a_quantile_aggregate_is_refused() {
let root = complete(AggregateKind::Quantile { quantile: 0.5 }, by(&["k0"]))(window());
let error = finalize(root, output_schema(&["q(v0)"], &["k0"])).unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
/// A tree with no aggregate at all is not this node's.
#[test]
fn a_tree_without_an_aggregate_is_refused() {
let error = finalize(window(), output_schema(&["v0"], &[])).unwrap_err();
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{error}"
);
}
/// The caller's schema has to describe the rows this node reports: one value column per field the
/// aggregate folds.
#[test]
fn the_output_schema_has_to_match_the_cut() {
let root = || complete(AggregateKind::Sum, by(&["k0"]))(window());
for output in [
// No value column where the aggregate folds one field.
output_schema(&[], &["k0"]),
// A value column per field the aggregate does not fold.
output_schema(&["sum(v0)", "sum(v1)"], &["k0"]),
] {
let error = finalize(root(), output).unwrap_err();
assert!(matches!(error, DataFusionError::Plan(_)), "{error}");
}
assert!(finalize(root(), output_schema(&["sum(v0)"], &["k0"])).is_ok());
}
/// The two halves of a split name their own sources, and neither reuses the other's.
///
/// The prefix still reads the one raw scan at execution source `0` and declares no grid - it is raw
/// samples. The suffix is a tree of its own, built at the cut, so its one input is source `0` of
/// *its* execution and carries the cut aggregate's evaluated output, on the same grid the node was
/// bound to: the merged values enter there, as the aggregate's own output shape.
#[test]
fn the_prefix_and_the_suffix_each_name_their_own_source_zero() {
let root = complete(AggregateKind::Sum, by(&["k0"]))(window());
let node = finalize(root, output_schema(&["sum(v0)"], &["k0"])).unwrap();
// The prefix is what the producer runs: one raw input on source 0, and no evaluation grid.
fn inputs(node: &NativeNode, found: &mut Vec<(usize, Option<Arc<EvalGrid>>)>) {
match &node.expr {
NativeExpr::Input(input) => found.push((
input.source_index,
input
.output
.samples()
.cloned()
.and_then(|schema| schema.eval_grid.clone()),
)),
NativeExpr::Scalar(_) | NativeExpr::EvalTime { .. } => {}
NativeExpr::Instant { input, .. }
| NativeExpr::Window { input, .. }
| NativeExpr::Merge { input }
| NativeExpr::Map { input, .. }
| NativeExpr::Timestamp { input }
| NativeExpr::Aggregate { input, .. }
| NativeExpr::Vector { input, .. }
| NativeExpr::VectorOf { input } => inputs(input, found),
NativeExpr::Binary { left, right, .. }
| NativeExpr::Set { left, right, .. }
| NativeExpr::Info {
base: left,
info: right,
..
} => {
inputs(left, found);
inputs(right, found);
}
NativeExpr::ParameterizedMap {
input, parameters, ..
} => {
inputs(input, found);
for parameter in parameters {
inputs(parameter, found);
}
}
}
}
let prefix = producer(&node).tree().clone();
let mut raw_inputs = Vec::new();
inputs(prefix.root(), &mut raw_inputs);
let [raw_input] = raw_inputs.as_slice() else {
panic!("the prefix reads one raw input, got {raw_inputs:?}");
};
assert_eq!(0, raw_input.0, "the prefix reads execution source 0");
assert!(
raw_input.1.is_none(),
"the prefix's input is raw samples, so it has no grid: {raw_input:?}"
);
// The suffix is its own tree: one input, named by its own execution's source 0, evaluated on
// the grid the node is bound to.
let mut suffix_inputs = Vec::new();
inputs(node.suffix.root(), &mut suffix_inputs);
let [suffix_input] = suffix_inputs.as_slice() else {
panic!("the suffix reads one input, got {suffix_inputs:?}");
};
assert_eq!(
0, suffix_input.0,
"the suffix's merged values are source 0 of its own execution"
);
let grid = suffix_input
.1
.as_ref()
.expect("the merged values are evaluated, so they are on a grid");
assert_eq!(
node.binding.grid.spec(),
grid.spec(),
"the suffix runs on the grid the node is bound to"
);
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+2 -10
View File
@@ -29,9 +29,8 @@ use store_api::region_engine::RawLabelSource;
use table::table::scan::{RegionScanExec, SeriesScanExec};
use crate::extension_plan::{
Absent, AggregateFinalize, EmptyMetric, FusedSeries, HistogramFold, InstantManipulate,
RangeManipulate, RawSeriesSource, ScalarCalculate, SeriesDivide, SeriesNormalize,
UnionDistinctOn,
Absent, EmptyMetric, FusedSeries, HistogramFold, InstantManipulate, RangeManipulate,
RawSeriesSource, ScalarCalculate, SeriesDivide, SeriesNormalize, UnionDistinctOn,
};
pub struct PromExtensionPlanner;
@@ -85,13 +84,6 @@ impl ExtensionPlanner for PromExtensionPlanner {
session.config_options().execution.target_partitions,
)
.map(Some)
} else if let Some(node) = node.as_any().downcast_ref::<AggregateFinalize>() {
// The merge runs where every partition's state arrives, so its own input is the plan
// that exports them.
let Some(input) = physical_inputs.first() else {
return Ok(None);
};
Ok(Some(node.to_execution_plan(Arc::clone(input))?))
} else {
Ok(None)
}
@@ -22,8 +22,8 @@
//! Chains are transported intact to owners with their input-partition ID, followed by one charged
//! normal-EOF message per live output from each completed input. Owners group exact identities and
//! may publish an identity as soon as every input has contributed its final chain or normal EOF. The
//! frame is the child's own: evaluated chunks are decoded only to read routing labels, and outputs
//! publish the child's declared layout.
//! frame is the child's own: evaluated chunks are decoded only to read routing labels; aggregate-state
//! batches are schema-checked and read only for routing labels. Outputs publish the child's layout.
//!
//! # Routing
//!
@@ -97,11 +97,9 @@
//! records the read outcome, then sender closure lets outputs detect missing EOF or report failure. If the child errors, the failing task records it before cancelling its peers; sender closure makes
//! every output report that recorded error instead of a clean end.
//!
//! A queue closing is not by itself that end: it also closes when the read is stopped - aborted, or
//! panicked - and a stopped read is missing exactly what it did not route. The read therefore records
//! how it ended, and an output whose queue has closed reads that record, so only a read that ended on
//! its own terms ends an output and anything else is an error saying the producer stopped
//! unexpectedly.
//! Shared execution records the read outcome before producer senders close, so queue closure together
//! with that shared outcome distinguishes normal completion from a stopped or failed read. A stopped
//! read is missing exactly what it did not route and is reported as an error rather than a clean end.
//!
//! An output stream owns the read, not the plan it was executed from: DataFusion's
//! `execute_stream_partitioned` takes the plan by value and keeps only the streams, so a read tied to
@@ -136,8 +134,9 @@ use datafusion::physical_plan::{
DisplayAs, DisplayFormatType, Distribution, ExecutionPlan, InputDistributionRequirements,
Partitioning, PlanProperties, RecordBatchStream, SendableRecordBatchStream,
};
use datatypes::series::aggregate_state::AggregateStateSchema;
use datatypes::series::frame::{
FrameContent, FrameLabelMode, LabelDecodeSpec, SeriesFrameDeclaration,
FrameContent, FrameLabelMode, LabelDecodeSpec, SeriesFrameDeclaration, SeriesFrameDecoder,
};
use datatypes::series::raw::RawSeriesChunk;
use datatypes::series::samples::{Series, SeriesSchema};
@@ -157,6 +156,8 @@ use datatypes::series::samples::DenseSamplesRef;
pub enum SeriesRouting {
/// Hash the canonical label set.
CanonicalLabels,
/// Hash the canonical label set without these labels, including an excluded metric name.
CanonicalLabelsExcept(Arc<[Arc<str>]>),
/// Hash the exact PromQL matching key, so both sides of a vector operation meet.
Matching(crate::vector_match::MatchingLabels),
/// Route raw frames by their complete stored primary key.
@@ -200,6 +201,7 @@ pub struct SeriesRepartitionExec {
routing_policy: SeriesRouting,
identity_completion: IdentityCompletion,
content: FrameContent,
state: Option<Arc<AggregateStateSchema>>,
ordering: SeriesOrdering,
/// Started by the first `execute` call, since the output partitions share one read of the child.
shared: Mutex<Option<Arc<SharedExecution>>>,
@@ -229,6 +231,7 @@ impl SeriesRepartitionExec {
let policy = match declaration.content() {
FrameContent::Evaluated => SeriesRouting::CanonicalLabels,
FrameContent::Raw => SeriesRouting::RawPrimaryKey,
FrameContent::AggregateState => SeriesRouting::CanonicalLabels,
};
Self::try_new_with_routing(input, outputs, policy)
}
@@ -249,6 +252,12 @@ impl SeriesRepartitionExec {
let policy = match declaration.content() {
FrameContent::Evaluated => SeriesRouting::CanonicalLabels,
FrameContent::Raw => SeriesRouting::RawPrimaryKey,
FrameContent::AggregateState => {
return Err(DataFusionError::NotImplemented(
"SeriesRepartition does not route aggregate state; use StateRepartitionExec"
.into(),
));
}
};
Self::try_new_with_completion(input, outputs, policy, IdentityCompletion::CompleteDisjoint)
}
@@ -296,11 +305,23 @@ impl SeriesRepartitionExec {
"SeriesRepartition evaluated frames cannot use RawPrimaryKey routing".into(),
));
}
(FrameContent::Raw, SeriesRouting::CanonicalLabels) => {
(
FrameContent::Raw,
SeriesRouting::CanonicalLabels | SeriesRouting::CanonicalLabelsExcept(_),
) => {
return Err(DataFusionError::Plan(
"SeriesRepartition raw frames cannot use CanonicalLabels routing".into(),
));
}
(
FrameContent::AggregateState,
SeriesRouting::RawPrimaryKey | SeriesRouting::Matching(_),
) => {
return Err(DataFusionError::Plan(
"SeriesRepartition aggregate-state frames require canonical-label routing"
.into(),
));
}
_ => {}
}
let series_schema = Arc::new(declaration.series().clone());
@@ -321,17 +342,24 @@ impl SeriesRepartitionExec {
&store_api::region_engine::NO_BOUND_LABEL_SOURCE,
ordering,
)?,
FrameContent::AggregateState => input.schema(),
};
// The input is re-declared from what it says about itself, not from what this node
// publishes: the two differ whenever the child carries its labels in its keys.
let rebuilt_input = common_query::series_frame::frame_schema_with(
&series_schema,
declaration.labels(),
declaration.label_decode(),
ordering,
)
.map_err(|error| DataFusionError::Plan(error.to_string()))?;
if rebuilt_input != input.schema() {
let rebuilt_input = if content == FrameContent::AggregateState {
None
} else {
Some(
common_query::series_frame::frame_schema_with(
&series_schema,
declaration.labels(),
declaration.label_decode(),
ordering,
)
.map_err(|error| DataFusionError::Plan(error.to_string()))?,
)
};
if rebuilt_input.is_some_and(|rebuilt| rebuilt != input.schema()) {
return Err(DataFusionError::Plan(
"SeriesRepartition input frame carries a schema this node cannot re-declare"
.to_string(),
@@ -347,6 +375,7 @@ impl SeriesRepartitionExec {
)
.with_evaluation_type(EvaluationType::Eager)
.with_scheduling_type(SchedulingType::Cooperative);
let state = declaration.aggregate_state().cloned().map(Arc::new);
let properties = Arc::new(properties);
Ok(Self {
input,
@@ -355,6 +384,7 @@ impl SeriesRepartitionExec {
routing_policy,
identity_completion,
content,
state,
ordering,
shared: Mutex::new(None),
properties,
@@ -477,6 +507,7 @@ impl ExecutionPlan for SeriesRepartitionExec {
&self.input,
self.routing.clone(),
Arc::clone(&self.series_schema),
self.state.clone(),
&context,
)?);
}
@@ -488,6 +519,15 @@ impl ExecutionPlan for SeriesRepartitionExec {
))
})?;
// Take the queue and its claim on the read together.
if self.content == FrameContent::AggregateState {
return Ok(Box::pin(StateBatchOutputStream {
schema: self.input.schema(),
receiver,
lease,
retained: None,
reported: false,
}));
}
let inner = RepartitionStream::new(
receiver,
lease,
@@ -513,6 +553,7 @@ impl ExecutionPlan for SeriesRepartitionExec {
};
pack_raw_series(Box::pin(raw))?
}
FrameContent::AggregateState => unreachable!("aggregate state output handled above"),
};
let schema = output.schema();
let output = Box::pin(PackedHandoffStream {
@@ -583,6 +624,7 @@ impl SharedExecution {
input: &Arc<dyn ExecutionPlan>,
routing: Routing,
series_schema: Arc<SeriesSchema>,
state: Option<Arc<AggregateStateSchema>>,
context: &Arc<TaskContext>,
) -> DataFusionResult<Arc<Self>> {
// What the child carries is what it declares, not what the policy suggests: a raw frame
@@ -611,6 +653,12 @@ impl SharedExecution {
.map_err(|error| DataFusionError::External(Box::new(error)))?;
RoutedInput::Raw(stream)
}
FrameContent::AggregateState => {
let state = state.as_ref().ok_or_else(|| {
DataFusionError::Plan("aggregate state declaration is missing".into())
})?;
RoutedInput::State(frames, Arc::clone(state), Arc::clone(&series_schema))
}
});
}
let mut senders = Vec::with_capacity(routing.outputs);
@@ -1002,6 +1050,200 @@ impl Inputs {
}
}
/// Upper bound for transient arrays allocated by slicing one aggregate-state row. It counts each
/// schema node's concrete array object, two `usize` wrapper words, and one reference per child.
fn state_slice_wrapper_bytes(
schema: &datafusion::arrow::datatypes::Schema,
) -> DataFusionResult<usize> {
use datafusion::arrow::array::{
ArrayRef, BinaryArray, BooleanArray, ListArray, PrimitiveArray, StringArray, StructArray,
};
use datafusion::arrow::datatypes::{DataType, Float64Type, UInt64Type};
fn node(data_type: &DataType) -> DataFusionResult<usize> {
let children = match data_type {
DataType::Struct(fields) => fields
.iter()
.map(|field| node(field.data_type()))
.collect::<DataFusionResult<Vec<_>>>()?,
DataType::List(_) => Vec::new(),
_ => Vec::new(),
};
let object = match data_type {
DataType::Struct(_) => std::mem::size_of::<StructArray>(),
DataType::List(_) => std::mem::size_of::<ListArray>(),
DataType::Utf8 | DataType::LargeUtf8 => std::mem::size_of::<StringArray>(),
DataType::Binary | DataType::LargeBinary => std::mem::size_of::<BinaryArray>(),
DataType::Boolean => std::mem::size_of::<BooleanArray>(),
DataType::Float64 => std::mem::size_of::<PrimitiveArray<Float64Type>>(),
DataType::UInt64 => std::mem::size_of::<PrimitiveArray<UInt64Type>>(),
// Frame schema types outside this list are bounded by the largest enumerated object.
_ => [
std::mem::size_of::<StructArray>(),
std::mem::size_of::<ListArray>(),
std::mem::size_of::<StringArray>(),
std::mem::size_of::<BinaryArray>(),
std::mem::size_of::<BooleanArray>(),
std::mem::size_of::<PrimitiveArray<Float64Type>>(),
std::mem::size_of::<PrimitiveArray<UInt64Type>>(),
]
.into_iter()
.max()
.unwrap(),
};
object
.checked_add(2 * std::mem::size_of::<usize>())
.and_then(|bytes| bytes.checked_add(children.len() * std::mem::size_of::<ArrayRef>()))
.and_then(|bytes| children.into_iter().try_fold(bytes, usize::checked_add))
.ok_or_else(|| {
DataFusionError::Execution("aggregate-state slice wrapper size overflow".into())
})
}
let arrays = schema.fields().iter().try_fold(0usize, |bytes, field| {
bytes.checked_add(node(field.data_type())?).ok_or_else(|| {
DataFusionError::Execution("aggregate-state slice wrapper size overflow".into())
})
})?;
// `RecordBatch::slice` also allocates the vector it collects the sliced columns into.
arrays
.checked_add(
schema
.fields()
.len()
.checked_mul(std::mem::size_of::<ArrayRef>())
.ok_or_else(|| {
DataFusionError::Execution("aggregate-state slice wrapper size overflow".into())
})?,
)
.ok_or_else(|| {
DataFusionError::Execution("aggregate-state slice wrapper size overflow".into())
})
}
/// Routes state rows without materializing Rust accumulator objects or rebuilding Arrow columns.
async fn read_state_batches(
stream: &mut SendableRecordBatchStream,
_partition: usize,
routing: &Routing,
senders: &[mpsc::UnboundedSender<Queued>],
read_ahead: &MemoryReservation,
state: &AggregateStateSchema,
series: &SeriesSchema,
) -> Stop {
use datatypes::series::frame::FrameLabelMode;
let decoder = match SeriesFrameDecoder::try_new_aggregate_state(
state,
series,
FrameLabelMode::Explicit,
) {
Ok(decoder) => decoder,
Err(error) => return Stop::Failed(Arc::new(DataFusionError::Execution(error.to_string()))),
};
let expected_schema = match datatypes::series::frame::aggregate_state_frame_schema(
state,
series,
FrameLabelMode::Explicit,
) {
Ok(schema) => schema,
Err(error) => return Stop::Failed(Arc::new(DataFusionError::Execution(error.to_string()))),
};
while let Some(next) = stream.next().await {
let batch = match next {
Ok(batch) => batch,
Err(error) => {
return Stop::Failed(Arc::new(DataFusionError::External(Box::new(error))));
}
};
if batch.schema().as_ref() != expected_schema.as_ref() {
return Stop::Failed(Arc::new(DataFusionError::Execution(
"aggregate state frame schema mismatch".into(),
)));
}
let charge = read_ahead.new_empty();
// Charge the input batch's retained buffers once. List slices still reference their entire
// values array, so a slice-size estimate cannot safely charge each queued row instead.
let batch_bytes = std::mem::size_of::<RecordBatch>()
.saturating_add(
batch
.num_columns()
.saturating_mul(std::mem::size_of::<datafusion::arrow::array::ArrayRef>()),
)
.saturating_add(batch.get_array_memory_size());
if let Err(error) = charge.try_grow(batch_bytes) {
return Stop::Failed(Arc::new(error));
}
let labels = match decoder.decode_aggregate_state_labels(&batch) {
Ok(labels) => labels,
Err(error) => {
return Stop::Failed(Arc::new(DataFusionError::Execution(error.to_string())));
}
};
let charge = Arc::new(charge);
let batch = Arc::new(batch);
if routing.outputs == 1 {
// Every row has the same owner. Keep the frame packed so a downstream router reads
// its shared payload once rather than receiving one parent-retaining slice per row.
if !labels.is_empty() && !senders[0].is_closed() {
let metadata = read_ahead.new_empty();
// Cloning the batch for the output allocates a new column-reference vector.
let wrapper_bytes = std::mem::size_of::<Queued>()
.saturating_add(std::mem::size_of::<RecordBatch>())
.saturating_add(
batch.num_columns().saturating_mul(std::mem::size_of::<
datafusion::arrow::array::ArrayRef,
>()),
);
if let Err(error) = metadata.try_grow(wrapper_bytes) {
return Stop::Failed(Arc::new(error));
}
if let Err(error) = senders[0].send(Queued::StateBatch {
batch,
row: None,
_charge: charge,
_metadata: metadata,
}) {
drop(error.0);
}
}
tokio::task::coop::consume_budget().await;
continue;
}
// Each queued row receives a reservation for the schema-based slice-wrapper upper bound.
let slice_charge = match state_slice_wrapper_bytes(batch.schema().as_ref()) {
Ok(bytes) => bytes,
Err(error) => return Stop::Failed(Arc::new(error)),
};
let slice_metadata_bytes = match std::mem::size_of::<Queued>().checked_add(slice_charge) {
Some(bytes) => bytes,
None => {
return Stop::Failed(Arc::new(DataFusionError::Execution(
"aggregate-state slice wrapper size overflow".into(),
)));
}
};
for (index, metric) in labels.iter().enumerate() {
let output = routing.output_labels(metric);
if senders[output].is_closed() {
continue;
}
// All slices retain the same parent buffers; a shared reservation charges them once.
let metadata = read_ahead.new_empty();
if let Err(error) = metadata.try_grow(slice_metadata_bytes) {
return Stop::Failed(Arc::new(error));
}
if let Err(error) = senders[output].send(Queued::StateBatch {
batch: Arc::clone(&batch),
row: Some(index),
_charge: Arc::clone(&charge),
_metadata: metadata,
}) {
drop(error.0);
}
}
tokio::task::coop::consume_budget().await;
}
Stop::Ended
}
/// One input partition's task: reads its own child, assembles chains, and sends each complete chain to
/// its canonical owner. Output queues are unbounded, so sending does not wait for an owner to poll;
/// queued chains remain charged to the pool until consumed or dropped. Each task records its outcome
@@ -1023,6 +1265,18 @@ async fn route_partition(
RoutedInput::Raw(stream) => {
read_raw_chains(stream, partition, &routing, &senders, &read_ahead).await
}
RoutedInput::State(stream, state, series) => {
read_state_batches(
stream,
partition,
&routing,
&senders,
&read_ahead,
state,
series,
)
.await
}
};
if matches!(&stop, Stop::Ended) {
for sender in &senders {
@@ -1498,6 +1752,11 @@ enum Ending {
enum RoutedInput {
Evaluated(SendableEvaluatedSeriesStream),
Raw(SendableRawSeriesStream),
State(
SendableRecordBatchStream,
Arc<AggregateStateSchema>,
Arc<SeriesSchema>,
),
}
enum RoutedChain {
@@ -1516,6 +1775,13 @@ enum Queued {
input: usize,
_metadata: MemoryReservation,
},
StateBatch {
batch: Arc<RecordBatch>,
// None forwards the entire parent; Some selects one row for multi-output routing.
row: Option<usize>,
_charge: Arc<MemoryReservation>,
_metadata: MemoryReservation,
},
}
type ExactIdentity = (Option<Arc<str>>, Vec<(Arc<str>, Arc<str>)>);
@@ -1753,6 +2019,88 @@ struct ReadyCharges {
}
/// One owner processes identities as soon as every input has closed or contributed its final chain.
struct StateBatchOutputStream {
schema: SchemaRef,
receiver: mpsc::UnboundedReceiver<Queued>,
lease: OutputLease,
retained: Option<(Arc<MemoryReservation>, MemoryReservation)>,
reported: bool,
}
impl RecordBatchStream for StateBatchOutputStream {
fn schema(&self) -> SchemaRef {
Arc::clone(&self.schema)
}
}
impl Stream for StateBatchOutputStream {
type Item = DataFusionResult<RecordBatch>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
if self.reported {
return Poll::Ready(None);
}
self.retained = None;
loop {
if let Ending::Failed(error) = self.lease.shared.ending() {
self.reported = true;
self.receiver.close();
while self.receiver.try_recv().is_ok() {}
self.retained = None;
return Poll::Ready(Some(Err(DataFusionError::from(&error))));
}
match self.receiver.poll_recv(cx) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Some(Queued::StateBatch {
batch,
row,
_charge,
_metadata,
})) => {
// Keep the router reservation through the next poll so Final can admit the
// returned batch's retained component arrays before releasing router charge.
self.retained = Some((_charge, _metadata));
let batch = match row {
Some(row) => batch.slice(row, 1),
None => batch.as_ref().clone(),
};
return Poll::Ready(Some(Ok(batch)));
}
Poll::Ready(Some(Queued::Eof { input, _metadata })) => {
// The shared Ending state and channel closure establish stream completion;
// this message is only a queue-order barrier for the routed state stream.
let _ = input;
drop(_metadata);
}
Poll::Ready(Some(Queued::Chain { .. })) => {
let error = DataFusionError::Internal(
"series router received a non-state message".into(),
);
self.lease.fail(Arc::new(DataFusionError::Internal(
"series router received a non-state message".into(),
)));
self.reported = true;
self.receiver.close();
while self.receiver.try_recv().is_ok() {}
self.retained = None;
return Poll::Ready(Some(Err(error)));
}
Poll::Ready(None) => {
self.reported = true;
self.retained = None;
return match self.lease.ending() {
Some(Err(error)) => {
Poll::Ready(Some(Err(DataFusionError::External(Box::new(error)))))
}
Some(Ok(_)) => unreachable!(),
None => Poll::Ready(None),
};
}
}
}
}
}
struct RepartitionStream {
receiver: mpsc::UnboundedReceiver<Queued>,
lease: OutputLease,
@@ -2099,6 +2447,11 @@ impl Stream for RepartitionStream {
return Poll::Pending;
}
match this.receiver.poll_recv(cx) {
Poll::Ready(Some(Queued::StateBatch { .. })) => {
this.lease.fail(Arc::new(DataFusionError::Internal(
"non-state output received an aggregate-state batch".into(),
)));
}
Poll::Ready(Some(Queued::Chain {
input,
chain,
@@ -2296,11 +2649,14 @@ impl Routing {
let mut hasher = XxHash64::with_seed(self.seed);
match &self.policy {
SeriesRouting::CanonicalLabels => label_set(metric).hash(&mut hasher),
SeriesRouting::CanonicalLabelsExcept(excluded) => {
let mut labels = label_set(metric);
labels.retain(|(name, _)| !excluded.contains(name));
labels.hash(&mut hasher);
}
SeriesRouting::Matching(policy) => {
crate::vector_match::group_key(metric, policy).hash(&mut hasher)
}
// A raw frame routed by its stored key never reaches here: its destination is the
// stored key itself, not a label hash. Evaluated frames never use this policy.
SeriesRouting::RawPrimaryKey => {
debug_assert!(
false,
@@ -41,6 +41,7 @@ use std::time::Duration;
use datafusion::arrow::array::{Array, Float64Array, ListArray, StructArray, UInt32Array};
use datafusion::arrow::buffer::{Buffer, ScalarBuffer};
use datafusion::common::config::ConfigOptions;
use datafusion::datasource::memory::MemorySourceConfig;
use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool, UnboundedMemoryPool};
use datafusion::execution::runtime_env::RuntimeEnvBuilder;
use datafusion::physical_optimizer::PhysicalOptimizerRule;
@@ -53,6 +54,9 @@ use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
use datafusion::prelude::SessionContext;
use datatypes::data_type::ConcreteDataType;
use datatypes::series::Label;
use datatypes::series::aggregate_state::{
AggregateStateKind, AggregateStateSchema, AggregateStateValues, StateComponent, StateSeries,
};
use datatypes::series::frame::{
FrameLabelMode, LABEL_DECODE_VERSION, LabelDecodeColumn, LabelDecodeEncoding, LabelDecodeTag,
};
@@ -977,6 +981,617 @@ async fn raw_primary_key_still_groups_a_key_labelled_input_by_its_key() {
);
}
#[test]
fn canonical_labels_except_routes_histogram_buckets_and_metadata_together() {
let routing = Routing::with_policy(
97,
SeriesRouting::CanonicalLabelsExcept(
["le", "__name__", "__type__", "__unit__"]
.into_iter()
.map(Arc::from)
.collect::<Vec<_>>()
.into(),
),
);
let group = |job: &str, le: &str, name: &str, unit: &str| {
SeriesLabels::try_new(
Some(Arc::from(name)),
vec![
Label {
name: Arc::from("job"),
value: Arc::from(job),
},
Label {
name: Arc::from("le"),
value: Arc::from(le),
},
Label {
name: Arc::from("__unit__"),
value: Arc::from(unit),
},
],
)
.unwrap()
};
let owner = routing.output(&group("api", "1", "m", "seconds"));
assert_eq!(owner, routing.output(&group("api", "+Inf", "n", "bytes")));
// Hash partitions can collide, so compare a set of distinct jobs, not an arbitrary pair.
assert!((0..100).any(|job| {
routing.output(&group(&format!("job-{job}"), "1", "m", "seconds")) != owner
}));
}
#[tokio::test]
async fn aggregate_state_router_slices_and_preserves_group_validity() {
use futures::TryStreamExt;
let series = schema();
let wire = Arc::new(
AggregateStateSchema::try_new(AggregateStateKind::Group, Vec::new())
.expect("Group has no components"),
);
// Two producers both contribute to the same group, including the same grid point. Routing
// must preserve both partial-state rows rather than treating a repeated point as a duplicate.
let mut batches = [vec![0, 1], vec![0, 2]]
.into_iter()
.map(|positions| {
let states = positions
.into_iter()
.map(|position| {
Ok((
StateSeries {
metric: labels("same"),
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(position, 1)],
)
.unwrap(),
states: vec![vec![None]],
},
true,
))
})
.collect::<Vec<_>>();
let mut frames = common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(states)),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap();
futures::executor::block_on(futures::future::poll_fn(|cx| {
match frames.as_mut().poll_next(cx) {
Poll::Ready(batch) => Poll::Ready(batch.expect("state batch is emitted")),
Poll::Pending => Poll::Pending,
}
}))
.unwrap()
})
.collect::<Vec<_>>();
let child_schema = batches[0].schema();
let child = MemorySourceConfig::try_new_exec(
&[vec![batches.remove(0)], vec![batches.remove(0)]],
child_schema,
None,
)
.unwrap();
let plan = Arc::new(SeriesRepartitionExec::try_new(child, 2).unwrap());
let pool = Arc::new(GreedyMemoryPool::new(8 << 20));
let context = task_context(&pool);
let outputs = futures::future::join_all((0..2).map(|partition| {
let plan = Arc::clone(&plan);
let context = Arc::clone(&context);
async move {
plan.execute(partition, context)
.unwrap()
.try_collect::<Vec<_>>()
.await
.unwrap()
}
}))
.await;
let rows = outputs
.into_iter()
.flatten()
.map(|batch| batch.num_rows())
.sum::<usize>();
assert_eq!(
rows, 4,
"both producers' rows are forwarded, including their overlapping point, without losing Group presence"
);
assert_eq!(pool.reserved(), 0, "terminal output releases state slices");
}
#[tokio::test]
async fn aggregate_state_slice_wrappers_remain_charged_until_terminal_cleanup() {
let grid = grid();
let fields = (0..16)
.map(|index| FieldSpec {
name: Arc::from(format!("slice_field_{index}")),
})
.collect::<Vec<_>>();
let series = Arc::new(SeriesSchema {
fields: fields.into(),
eval_grid: Some(Arc::clone(&grid)),
});
let wire = Arc::new(
AggregateStateSchema::try_new(
AggregateStateKind::Avg,
vec![
StateComponent::Value,
StateComponent::Mean,
StateComponent::Compensation,
StateComponent::Count,
StateComponent::IncrementalMean,
],
)
.unwrap(),
);
const ROWS: usize = 8;
let rows = (0..ROWS)
.map(|position| {
Ok((
StateSeries {
metric: labels("many-slice-wrappers"),
time: TimeAxis::from_grid_spans(
Arc::clone(&grid),
vec![GridSpan::new(position, 1)],
)
.unwrap(),
states: vec![vec![Some(AggregateStateValues::default())]; series.fields.len()],
},
true,
))
})
.collect::<Vec<_>>();
let mut frames = common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(rows)),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap();
let batch = frames
.next()
.await
.expect("state batch is emitted")
.expect("state batch packs successfully");
assert_eq!(ROWS, batch.num_rows());
let backing_bytes = std::mem::size_of::<RecordBatch>()
+ batch.num_columns() * std::mem::size_of::<datafusion::arrow::array::ArrayRef>()
+ batch.get_array_memory_size();
let slice_wrappers = state_slice_wrapper_bytes(batch.schema().as_ref()).unwrap();
// Hand-count the declared shape: 23 sliced array nodes; list values are not collected.
// There are 17 struct-child refs and six root-column refs.
use datafusion::arrow::array::{
ArrayRef, BinaryArray, BooleanArray, ListArray, StringArray, StructArray,
};
let node_bytes = std::mem::size_of::<StructArray>() * 2
+ std::mem::size_of::<ListArray>() * 18
+ std::mem::size_of::<BinaryArray>()
+ std::mem::size_of::<BooleanArray>()
+ std::mem::size_of::<StringArray>();
let expected_wrappers =
node_bytes + 23 * 2 * std::mem::size_of::<usize>() + 23 * std::mem::size_of::<ArrayRef>();
assert_eq!(
expected_wrappers, slice_wrappers,
"schema-shape upper bound"
);
let child_schema = batch.schema();
let child: Arc<dyn ExecutionPlan> =
MemorySourceConfig::try_new_exec(&[vec![batch]], child_schema.clone(), None).unwrap();
let budget =
backing_bytes + ROWS * (slice_wrappers + std::mem::size_of::<Queued>()) + (4 << 10);
let pool = Arc::new(GreedyMemoryPool::new(budget));
let context = task_context(&pool);
let shared = SharedExecution::start(
&child,
Routing::new(2),
Arc::clone(&series),
Some(Arc::clone(&wire)),
&context,
)
.unwrap();
let owner = Routing::new(2).output(&labels("many-slice-wrappers"));
let (receiver, lease) = SharedExecution::claim(&shared, owner).expect("claim output");
let mut output = StateBatchOutputStream {
schema: child_schema,
receiver,
lease,
retained: None,
reported: false,
};
wait_until("state producer queues every slice", || {
producer_stopped(&shared)
})
.await;
let expected_payload_and_wrappers = backing_bytes + ROWS * slice_wrappers;
assert!(
pool.reserved() >= expected_payload_and_wrappers,
"queued state slices retain one shared backing charge plus {ROWS} recursive wrapper charges: reserved {}, expected at least {expected_payload_and_wrappers}",
pool.reserved()
);
// Inspect queued rows directly: all carry the same Arc parent, proving the producer does not
// allocate a RecordBatch slice per row. The normal stream path slices only when polled.
let mut batches = Vec::with_capacity(ROWS);
let mut charges = Vec::with_capacity(ROWS);
for _ in 0..ROWS {
let message = output.receiver.recv().await.expect("a queued state row");
let message = match message {
Queued::StateBatch {
batch,
row,
_charge,
_metadata,
} => {
let row = row.expect("multi-output routing selects one row");
assert!(row < ROWS);
assert_eq!(1, batch.slice(row, 1).num_rows());
charges.push(_charge);
batch
}
_ => panic!("expected aggregate-state row"),
};
batches.push(message);
}
assert!(
batches
.windows(2)
.all(|pair| Arc::ptr_eq(&pair[0], &pair[1]))
);
assert!(
charges
.windows(2)
.all(|pair| Arc::ptr_eq(&pair[0], &pair[1])),
"every row shares the same parent reservation"
);
assert!(pool.reserved() >= backing_bytes);
drop(batches);
drop(charges);
assert!(
output.next().await.is_none(),
"state output drains to clean EOF"
);
drop(output);
drop(shared);
assert_eq!(
0,
pool.reserved(),
"terminal cleanup releases slice reservations"
);
}
/// Region exit forwards one packed frame, then Final routes its rows while retaining its parent
/// exactly once, even though Arrow's List slices still retain all of the component values.
#[tokio::test]
async fn chained_state_routers_share_parent_charge_for_multi_point_groups() {
const GROUPS: usize = 128;
const POINTS: usize = 60;
const OUTPUTS: usize = 2;
const COPIES: usize = 64;
let series = schema();
let wire = Arc::new(
AggregateStateSchema::try_new(
AggregateStateKind::Sum,
vec![StateComponent::Value, StateComponent::Compensation],
)
.unwrap(),
);
let rows = (0..GROUPS)
.map(|group| {
let states = (0..POINTS)
.map(|point| {
Some(AggregateStateValues {
value: (group * POINTS + point) as f64,
compensation: -(point as f64),
..Default::default()
})
})
.collect();
Ok((
StateSeries {
metric: labels(&format!("group-{group:03}")),
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(0, POINTS)],
)
.unwrap(),
states: vec![states],
},
true,
))
})
.collect::<Vec<_>>();
let frames = common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(rows)),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap();
let mut batches = frames.try_collect::<Vec<_>>().await.unwrap();
assert_eq!(1, batches.len(), "all groups share one packed parent");
let batch = batches.pop().unwrap();
assert_eq!(GROUPS, batch.num_rows());
let parent_payload = std::mem::size_of::<RecordBatch>()
+ batch.num_columns() * std::mem::size_of::<datafusion::arrow::array::ArrayRef>()
+ batch.get_array_memory_size();
let wrappers = state_slice_wrapper_bytes(batch.schema().as_ref()).unwrap();
let message = std::mem::size_of::<Queued>();
// There are GROUPS output wrappers, OUTPUTS EOF messages, and briefly another parent during
// handoff between routers. Once the producer stops, only one parent and those wrappers remain.
let overhead = GROUPS * (wrappers + message) + OUTPUTS * message;
let bound = parent_payload + overhead;
let handoff_overhead = parent_payload + 2 * message + (4 << 10);
let budget = bound + handoff_overhead;
let duplicated_components = COPIES * GROUPS * POINTS * 2 * std::mem::size_of::<f64>();
assert!(
budget < duplicated_components,
"64+ duplicated payloads must not fit"
);
let pool = Arc::new(GreedyMemoryPool::new(budget));
let context = task_context(&pool);
let child_schema = batch.schema();
let probe_child =
MemorySourceConfig::try_new_exec(&[vec![batch.clone()]], Arc::clone(&child_schema), None)
.unwrap();
let probe = Arc::new(SeriesRepartitionExec::try_new(probe_child, 1).unwrap());
let mut probe_output = probe.execute(0, Arc::clone(&context)).unwrap();
let forwarded = probe_output.next().await.unwrap().unwrap();
assert_eq!(
GROUPS,
forwarded.num_rows(),
"region exit forwards the packed frame"
);
assert!(
Arc::ptr_eq(batch.column(3), forwarded.column(3)),
"the region exit does not slice the state column"
);
assert!(
probe_output.next().await.is_none(),
"one parent is one message"
);
drop(forwarded);
drop(probe_output);
drop(probe);
assert_eq!(0, pool.reserved(), "the region exit releases its charge");
let child = MemorySourceConfig::try_new_exec(&[vec![batch]], child_schema, None).unwrap();
let region = Arc::new(SeriesRepartitionExec::try_new(child, 1).unwrap());
let final_router = Arc::new(SeriesRepartitionExec::try_new(region.clone(), OUTPUTS).unwrap());
let mut outputs = (0..OUTPUTS)
.map(|output| final_router.execute(output, Arc::clone(&context)).unwrap())
.collect::<Vec<_>>();
let final_shared = execution(&final_router);
wait_until("the Final router queues its rows", || {
!matches!(final_shared.ending(), Ending::Pending)
})
.await;
let region_shared = execution(&region);
assert!(matches!(region_shared.ending(), Ending::Finished));
assert!(
matches!(final_shared.ending(), Ending::Finished),
"the bounded Final router must finish: {:?}",
final_shared.ending()
);
let observed = pool.reserved();
assert!(
observed >= parent_payload,
"the queued slices retain their parent"
);
assert!(
observed <= bound,
"queued rows must share their parent: observed {observed}, parent {parent_payload}, \
overhead {overhead}, bound {bound}"
);
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(&wire, &series, FrameLabelMode::Explicit)
.unwrap();
let mut seen = vec![false; GROUPS];
let mut owner_counts = [0; OUTPUTS];
for (owner, output) in outputs.iter_mut().enumerate() {
while let Some(batch) = output.next().await {
let batch = batch.expect("the Final router must not exhaust the pool");
assert_eq!(
1,
batch.num_rows(),
"multi-output routing still splits rows"
);
for row in decoder.decode_aggregate_state(&batch).unwrap() {
assert!(row.is_last());
let group = row.metric().tags()[0]
.value
.strip_prefix("group-")
.unwrap()
.parse::<usize>()
.unwrap();
assert!(!seen[group], "each group is routed exactly once");
assert_eq!(owner, final_router.routing.output_labels(row.metric()));
let row = row.to_state_series().unwrap();
assert_eq!(POINTS, row.time.len());
for (point, state) in row.states[0].iter().enumerate() {
let state = state.unwrap();
assert_eq!((group * POINTS + point) as f64, state.value);
assert_eq!(-(point as f64), state.compensation);
}
seen[group] = true;
owner_counts[owner] += 1;
}
}
}
assert!(seen.into_iter().all(|seen| seen));
assert!(owner_counts.into_iter().all(|count| count > 0));
drop(outputs);
drop(final_shared);
drop(region_shared);
drop(final_router);
drop(region);
assert_eq!(
0,
pool.reserved(),
"terminal cleanup releases the parent and wrappers"
);
}
#[tokio::test]
async fn aggregate_state_router_holds_charge_for_batch_handoff_until_eof() {
let series = schema();
let wire = Arc::new(
AggregateStateSchema::try_new(AggregateStateKind::Group, Vec::new())
.expect("Group has no components"),
);
let state = StateSeries {
metric: labels("handoff"),
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(0, 1)],
)
.unwrap(),
states: vec![vec![None]],
};
let mut frames = common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(vec![Ok((state, true))])),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap();
let batch = futures::executor::block_on(futures::future::poll_fn(|cx| {
match frames.as_mut().poll_next(cx) {
Poll::Ready(batch) => Poll::Ready(batch.expect("state batch is emitted")),
Poll::Pending => Poll::Pending,
}
}))
.unwrap();
let child_schema = batch.schema();
let child = MemorySourceConfig::try_new_exec(&[vec![batch]], child_schema, None).unwrap();
let plan = Arc::new(SeriesRepartitionExec::try_new(child, 1).unwrap());
let pool = Arc::new(GreedyMemoryPool::new(8 << 20));
let context = task_context(&pool);
let mut output = plan.execute(0, context).unwrap();
let returned = futures::future::poll_fn(|cx| match output.as_mut().poll_next(cx) {
Poll::Ready(Some(batch)) => Poll::Ready(batch),
Poll::Ready(None) => panic!("state router emitted no batch"),
Poll::Pending => Poll::Pending,
})
.await
.unwrap();
assert_eq!(1, returned.num_rows());
assert!(
pool.reserved() > 0,
"router memory remains charged while the returned batch has not been admitted downstream"
);
assert!(
futures::future::poll_fn(|cx| output.as_mut().poll_next(cx))
.await
.is_none(),
"state router reaches EOF after its one batch"
);
assert_eq!(
0,
pool.reserved(),
"router reservation is released at EOF while output, returned batch, and plan remain alive"
);
assert_eq!(1, returned.num_rows());
}
#[tokio::test]
async fn aggregate_state_router_drop_handoff_releases_charge() {
let series = schema();
let wire =
Arc::new(AggregateStateSchema::try_new(AggregateStateKind::Group, Vec::new()).unwrap());
let state = StateSeries {
metric: labels("drop-handoff"),
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(0, 1)],
)
.unwrap(),
states: vec![vec![None]],
};
let batch = futures::executor::block_on(
common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(vec![Ok((state, true))])),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap()
.try_collect::<Vec<_>>(),
)
.unwrap()
.remove(0);
let child_schema = batch.schema();
let child = MemorySourceConfig::try_new_exec(&[vec![batch]], child_schema, None).unwrap();
let plan = Arc::new(SeriesRepartitionExec::try_new(child, 1).unwrap());
let pool = Arc::new(GreedyMemoryPool::new(8 << 20));
let context = task_context(&pool);
let mut output = plan.execute(0, context).unwrap();
let returned = output.next().await.expect("state batch").unwrap();
assert!(
pool.reserved() > 0,
"handoff remains charged before downstream admission"
);
drop(returned);
assert!(output.next().await.is_none(), "router reaches EOF");
drop(output);
assert_eq!(
0,
pool.reserved(),
"dropping an unadmitted slice releases all charges"
);
}
#[tokio::test]
async fn aggregate_state_router_cancelled_consumer_releases_queued_charge() {
let series = schema();
let wire =
Arc::new(AggregateStateSchema::try_new(AggregateStateKind::Group, Vec::new()).unwrap());
let rows = (0..8)
.map(|position| {
Ok((
StateSeries {
metric: labels("cancel"),
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(position, 1)],
)
.unwrap(),
states: vec![vec![None]],
},
true,
))
})
.collect::<Vec<_>>();
let batch = futures::executor::block_on(
common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(rows)),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap()
.try_collect::<Vec<_>>(),
)
.unwrap()
.remove(0);
let child_schema = batch.schema();
let child = MemorySourceConfig::try_new_exec(&[vec![batch]], child_schema, None).unwrap();
let plan = Arc::new(SeriesRepartitionExec::try_new(child, 1).unwrap());
let pool = Arc::new(GreedyMemoryPool::new(8 << 20));
let context = task_context(&pool);
let output = plan.execute(0, context).unwrap();
wait_until("state producer queues its state", || pool.reserved() > 0).await;
drop(output);
tokio::time::timeout(Duration::from_secs(2), async {
while pool.reserved() != 0 {
tokio::time::sleep(Duration::from_millis(1)).await;
}
})
.await
.expect("cancellation releases all queued charges");
assert_eq!(0, pool.reserved(), "cancelled consumer holds nothing");
}
/// The raw owner over `child` routing by an explicit policy.
fn raw_node_with_routing(
child: Arc<dyn ExecutionPlan>,
@@ -2741,8 +3356,14 @@ async fn eof_readies_two_groups_but_poll_merges_one_at_a_time() {
]]);
let pool = Arc::new(GreedyMemoryPool::new(1 << 20));
let context = task_context(&pool);
let shared =
SharedExecution::start(&child, Routing::new(1), Arc::clone(&schema()), &context).unwrap();
let shared = SharedExecution::start(
&child,
Routing::new(1),
Arc::clone(&schema()),
None,
&context,
)
.unwrap();
let (receiver, lease) = SharedExecution::claim(&shared, 0).expect("claim output");
let mut stream = RepartitionStream::new(
receiver,
@@ -4768,8 +5389,14 @@ async fn a_recorded_failure_reaches_every_output_once() {
let context = task_context(&pool);
// Claim internal queues so assertions observe retained chains, not encoded frame batches.
let child = plan.children()[0].clone();
let shared =
SharedExecution::start(&child, Routing::new(2), Arc::clone(&schema()), &context).unwrap();
let shared = SharedExecution::start(
&child,
Routing::new(2),
Arc::clone(&schema()),
None,
&context,
)
.unwrap();
let mut streams = Vec::new();
for partition in 0..2 {
let (receiver, lease) = SharedExecution::claim(&shared, partition).expect("output claim");
@@ -5972,8 +6599,14 @@ async fn internal_stream_releases_each_chunk_charge_only_when_polled() {
.map(|(series, is_last)| Step::Chunk(series, is_last))
.collect(),
]);
let shared =
SharedExecution::start(&child, Routing::new(1), Arc::clone(&schema()), &context).unwrap();
let shared = SharedExecution::start(
&child,
Routing::new(1),
Arc::clone(&schema()),
None,
&context,
)
.unwrap();
let (receiver, lease) = SharedExecution::claim(&shared, 0).expect("claim output");
let mut stream = RepartitionStream::new(
receiver,
@@ -6270,8 +6903,14 @@ async fn histogram_chain_slices_fit_a_budget_below_the_full_table_charge() {
);
let pool = Arc::new(GreedyMemoryPool::new(budget));
let context = task_context(&pool);
let shared =
SharedExecution::start(&child, Routing::new(1), Arc::clone(&schema()), &context).unwrap();
let shared = SharedExecution::start(
&child,
Routing::new(1),
Arc::clone(&schema()),
None,
&context,
)
.unwrap();
let (receiver, lease) = SharedExecution::claim(&shared, 0).expect("claim output");
let mut output = RepartitionStream::new(
receiver,
@@ -7332,3 +7971,108 @@ async fn a_raw_merge_conflict_fails_the_execution() {
.contains("duplicate raw timestamp has conflicting samples")
);
}
/// A one-output region exit hands the Final router one packed batch containing several group
/// labels. Each row must be dispatched by its own key, not by the first label in the batch.
#[tokio::test]
async fn histogram_state_router_dispatches_each_row_of_a_packed_batch() {
let series = schema();
let wire = Arc::new(AggregateStateSchema::try_new(AggregateStateKind::Group, vec![]).unwrap());
let policy = SeriesRouting::CanonicalLabelsExcept(
["le", "__name__", "__type__", "__unit__"]
.into_iter()
.map(Arc::from)
.collect::<Vec<_>>()
.into(),
);
let routing = Routing::with_policy(8, policy.clone());
let group = |job: &str, le: &str| {
SeriesLabels::try_new(
None,
vec![
Label {
name: Arc::from("job"),
value: Arc::from(job),
},
Label {
name: Arc::from("le"),
value: Arc::from(le),
},
],
)
.unwrap()
};
let first = group("api", "1");
let other_job = (0..100)
.map(|index| format!("other-{index}"))
.find(|job| routing.output(&group(job, "1")) != routing.output(&first))
.unwrap();
let groups = [
first,
group("api", "+Inf"),
group(&other_job, "1"),
group(&other_job, "+Inf"),
];
let input = groups
.iter()
.cloned()
.map(|metric| {
Ok((
StateSeries {
metric,
time: TimeAxis::from_grid_spans(
series.eval_grid.clone().unwrap(),
vec![GridSpan::new(0, 1)],
)
.unwrap(),
states: vec![vec![None]],
},
true,
))
})
.collect::<Vec<_>>();
let mut packed = common_query::series_frame::pack_aggregate_state(
Box::pin(futures::stream::iter(input)),
Arc::clone(&wire),
Arc::clone(&series),
)
.unwrap();
let batch = packed.next().await.unwrap().unwrap();
assert_eq!(
batch.num_rows(),
groups.len(),
"fixture must be one multi-row packed batch"
);
let child =
MemorySourceConfig::try_new_exec(&[vec![batch.clone()]], batch.schema(), None).unwrap();
let region = Arc::new(SeriesRepartitionExec::try_new(child, 1).unwrap());
let final_router =
Arc::new(SeriesRepartitionExec::try_new_with_routing(region, 8, policy).unwrap());
let context = task_context(&Arc::new(GreedyMemoryPool::new(8 << 20)));
let outputs = futures::future::join_all((0..8).map(|partition| {
let router = Arc::clone(&final_router);
let context = Arc::clone(&context);
async move {
router
.execute(partition, context)
.unwrap()
.try_collect::<Vec<_>>()
.await
.unwrap()
}
}))
.await;
let decoder =
SeriesFrameDecoder::try_new_aggregate_state(&wire, &series, FrameLabelMode::Explicit)
.unwrap();
let mut seen = 0;
for (partition, batches) in outputs.into_iter().enumerate() {
for batch in batches {
for metric in decoder.decode_aggregate_state_labels(&batch).unwrap() {
assert_eq!(partition, routing.output(&metric));
seen += 1;
}
}
}
assert_eq!(seen, groups.len());
}
+6 -1
View File
@@ -47,7 +47,12 @@ use crate::grid_series::{
};
use crate::map::validate_grid_series;
const BUCKET_LABEL: &str = "le";
pub(crate) const BUCKET_LABEL: &str = "le";
/// Histogram inputs that can produce the same output must meet at one partition. The kernel
/// still groups by metadata before removing it, so distinct histograms are not mixed together.
pub(crate) const HISTOGRAM_ROUTING_EXCLUDED_LABELS: [&str; 4] =
[BUCKET_LABEL, "__name__", "__type__", "__unit__"];
/// `histogram_quantile` or `histogram_fraction` with its constant arguments.
#[derive(Debug, Clone, Copy, serde::Serialize, serde::Deserialize)]
+53
View File
@@ -907,3 +907,56 @@ fn native_histogram_functions_refuse_a_series_that_does_not_fit_the_schema() {
let wider = fields(&["value", "other"]);
assert!(map_native_histogram(NativeHistogramFunction::Count, &wider, &input).is_err());
}
/// Metadata is part of histogram identity even though the outputs later drop it. Pooling the
/// buckets first would hide overlapping outputs and give wrong values for disjoint outputs.
#[test]
fn classic_histograms_keep_metadata_identity_until_after_evaluation() {
let schema = Arc::new(SeriesSchema {
fields: vec![FieldSpec {
name: Arc::from("value"),
}]
.into(),
eval_grid: Some(Arc::new(
EvalGrid::try_new(GridSpec::try_new(0, 1_000, 2).unwrap()).unwrap(),
)),
});
for metadata in ["__name__", "__type__", "__unit__"] {
let build = |second_point: usize| {
let mut buckets = Vec::new();
for (identity, point, first) in [("a", 0, 1.0_f64), ("b", second_point, 3.0_f64)] {
for (le, count) in [("1", first), ("+Inf", 4.0_f64)] {
let labels = Labels::from([
(metadata.to_string(), identity.to_string()),
("job".to_string(), "api".to_string()),
("le".to_string(), le.to_string()),
]);
buckets.push(
mixed_series(
&schema,
&labels,
&BTreeMap::from([(point, count.to_bits())]),
&BTreeMap::new(),
)
.unwrap(),
);
}
}
buckets
};
let disjoint =
evaluate_histograms(HistogramFunction::Quantile(0.5), &schema, &build(1)).unwrap();
let points = points_by_labels(disjoint);
let values = &points[&Labels::from([("job".to_string(), "api".to_string())])];
assert_eq!(f64::from_bits(values[&0]), 1.0, "{metadata}");
assert!(
(f64::from_bits(values[&1]) - 2.0 / 3.0).abs() < 1e-12,
"{metadata}"
);
let error = evaluate_histograms(HistogramFunction::Quantile(0.5), &schema, &build(0));
assert!(
error.is_err(),
"overlapping metadata-distinct outputs must collide: {metadata}"
);
}
}
+7 -7
View File
@@ -248,6 +248,13 @@ impl std::hash::Hash for LabelRewrite {
}
impl LabelRewrite {
/// The label this rewrite writes, which is the one whose routing key may change.
pub fn destination(&self) -> &str {
match &self.0 {
Change::Replace { dst, .. } | Change::Join { dst, .. } => dst,
}
}
/// `label_replace`'s change, failing on an invalid expression or an empty destination.
pub fn replace(dst: &str, replacement: &str, src: &str, regex: &str) -> Result<Self> {
let compiled = Regex::new(&format!("^(?s:{regex})$")).map_err(|_| {
@@ -321,13 +328,6 @@ impl LabelRewrite {
}
}
/// Destination label written by this rewrite.
pub fn destination(&self) -> &str {
match &self.0 {
Change::Replace { dst, .. } | Change::Join { dst, .. } => dst,
}
}
/// Changes `metric` the way the function changes one series.
pub fn apply(&self, metric: &mut SeriesLabels) -> Result<()> {
let mut value = String::new();
+151 -33
View File
@@ -26,22 +26,30 @@
//! caller that schedules scans and exchanges guarantees it, and a plan that ran a `Complete`
//! aggregate once per partition and then combined the answers would be combining values that were
//! each missing the other partitions' contributions. The stages that exist for that shape are
//! [`AggregateStage::Partial`] and [`AggregateStage::Final`], which this runtime does not execute
//! yet.
//! [`AggregateStage::Partial`] and [`AggregateStage::Final`], which carry kernel accumulator
//! state between executions.
use std::sync::Arc;
use datafusion::error::{DataFusionError, Result};
use datatypes::series::samples::{Series, SeriesSchema};
use crate::aggregate::{AggregateOp, FloatAggregator, Grouping};
use crate::aggregate::{AggregateOp, FloatAggregator, FloatMerger, Grouping};
use crate::evaluated::EvaluatedChunk;
use crate::grid_series::series_bytes;
use crate::map::validate_float_series;
use crate::native::execute::{NodeBuffer, evaluated_bytes};
use crate::native::plan::{
AggregateKind, AggregateStage, BoundAggregate, Grouping as BoundGrouping,
AggregateKind, AggregateStage, AggregateStateSchema, BoundAggregate, Grouping as BoundGrouping,
aggregate_state_binding,
};
use crate::native::state::{StateChunk, state_bytes, state_chunk_from_partial};
/// The output shape produced by an aggregate stage.
pub(crate) enum AggregateOutput {
Samples(Vec<Series>),
State(StateChunk),
}
/// One aggregate node's states, from the first chunk it is handed to the series it finishes with.
pub(crate) struct AggregateRuntime {
@@ -54,6 +62,8 @@ pub(crate) struct AggregateRuntime {
/// The fold. `None` once [`Self::compute`] has taken it, which is also what says the node has
/// nothing left to fold into.
accumulator: Option<FloatAggregator>,
merger: Option<FloatMerger>,
state_schema: Option<Arc<AggregateStateSchema>>,
/// Whether the fold has been finished and its series handed over. A node computes once: a second
/// pull finds the series it queued rather than a fold that ran again.
///
@@ -81,14 +91,6 @@ impl AggregateRuntime {
schema: Arc<SeriesSchema>,
buffer: &mut NodeBuffer,
) -> Result<Self> {
if op.stage != AggregateStage::Complete {
return Err(DataFusionError::NotImplemented(format!(
"the native runtime does not execute a `{:?}` aggregate yet: a `Partial` aggregate \
reports the state a later stage merges and a `Final` aggregate merges that state, \
and only `Complete` -- every contribution inside one execution -- is computed here",
op.stage
)));
}
let kind = match op.kind {
AggregateKind::Sum => AggregateOp::Sum,
AggregateKind::Count => AggregateOp::Count,
@@ -104,6 +106,30 @@ impl AggregateRuntime {
)));
}
};
let (state_kind, state_components) = aggregate_state_binding(op.kind).ok_or_else(|| {
DataFusionError::NotImplemented("aggregate kind has no mergeable state".into())
})?;
let state_schema = schema.eval_grid.as_ref().map(|grid| {
Arc::new(AggregateStateSchema {
group: op.group.clone(),
kind: state_kind,
components: state_components.into(),
accumulators: schema.fields.clone(),
grid: grid.clone(),
})
});
if op.stage == AggregateStage::Final {
let merger = FloatMerger::try_new(&schema, kind)?;
buffer.resize(merger.state_bytes())?;
return Ok(Self {
op,
schema,
accumulator: None,
merger: Some(merger),
state_schema,
computed: false,
});
}
let name_count = match &op.group {
BoundGrouping::By(labels) | BoundGrouping::Without(labels) => labels.len(),
};
@@ -128,6 +154,8 @@ impl AggregateRuntime {
op,
schema,
accumulator: Some(initial),
merger: None,
state_schema,
computed: false,
})
}
@@ -171,6 +199,57 @@ impl AggregateRuntime {
buffer.resize(accumulator.state_bytes())
}
/// Merges one state chunk into this final aggregate.
pub(crate) fn accumulate_state(
&mut self,
chunk: StateChunk,
buffer: &mut NodeBuffer,
pending: usize,
) -> Result<()> {
let expected = self.state_schema.as_ref().ok_or_else(|| {
DataFusionError::NotImplemented("this runtime does not merge aggregate state".into())
})?;
let StateChunk::Row(row) = chunk else {
return Err(DataFusionError::Internal(
"Final aggregate received producer state series".into(),
));
};
if row.fields.kind() != expected.kind
|| row.fields.components() != expected.components.as_ref()
{
return Err(DataFusionError::NotImplemented(
"this runtime does not merge that kind of state".into(),
));
}
if row
.time
.grid()
.is_none_or(|grid| grid.spec() != expected.grid.spec())
{
return Err(DataFusionError::Execution(
"aggregate state grid mismatch".into(),
));
}
if row.fields.len() != self.schema.fields.len() {
return Err(DataFusionError::Execution(
"aggregate state field count mismatch".into(),
));
}
let description = self.describe();
let merger = self.merger.as_mut().ok_or_else(|| no_fold(&description))?;
merger.merge_row(&row, |bytes| buffer.resize(pending.saturating_add(bytes)))?;
drop(row);
buffer.resize(merger.state_bytes())
}
/// Transient bytes required to consume a state chunk.
pub(crate) fn pending_state(&self, chunk: &StateChunk) -> usize {
match chunk {
StateChunk::Series { .. } => state_bytes(chunk),
StateChunk::Row(row) => crate::native::state::state_row_bytes(row),
}
}
/// The number of retained groups, for budgeting the output queue backing storage.
pub(crate) fn groups(&self) -> usize {
self.accumulator.as_ref().map_or(0, FloatAggregator::groups)
@@ -185,34 +264,73 @@ impl AggregateRuntime {
&mut self,
buffer: &mut NodeBuffer,
queue_bytes: usize,
) -> Result<Option<Vec<Series>>> {
) -> Result<Option<AggregateOutput>> {
if self.computed {
return Ok(None);
}
if self.accumulator.is_none() {
return Err(no_fold(&self.describe()));
}
self.computed = true;
let output = self
.accumulator
.take()
.ok_or_else(|| no_fold(&self.describe()))?
.finish_budgeted(|total| buffer.resize(total.saturating_add(queue_bytes)))?;
let output_bytes = output.iter().fold(0usize, |bytes, series| {
bytes.saturating_add(series_bytes(series))
});
let output_vec_bytes = output
.capacity()
.saturating_mul(std::mem::size_of::<Series>());
buffer.resize(
output_bytes
.saturating_add(output_vec_bytes)
.saturating_add(queue_bytes),
)?;
Ok(Some(output))
if let Some(accumulator) = self.accumulator.take() {
return match self.op.stage {
AggregateStage::Complete => {
let output = accumulator.finish_budgeted(|total| {
buffer.resize(total.saturating_add(queue_bytes))
})?;
let output_vec_bytes = output
.capacity()
.saturating_mul(std::mem::size_of::<Series>());
charge_samples(buffer, &output, output_vec_bytes, queue_bytes)?;
Ok(Some(AggregateOutput::Samples(output)))
}
AggregateStage::Partial => {
let state_schema = self
.state_schema
.clone()
.ok_or_else(|| no_fold(&self.describe()))?;
// The fold builds the chunk's own representation, so handing it over costs no
// second copy and no second charge: what was admitted for the groups is what
// the chunk now holds.
let partial = accumulator
.finish_partial_budgeted(|bytes| buffer.resize(bytes + queue_bytes))?;
let chunk = state_chunk_from_partial(partial, state_schema)?;
Ok(Some(AggregateOutput::State(chunk)))
}
AggregateStage::Final => Err(no_fold(&self.describe())),
};
}
if let Some(merger) = self.merger.take() {
let output =
merger.finish_budgeted(|total| buffer.resize(total.saturating_add(queue_bytes)))?;
let output_vec_bytes = output
.capacity()
.saturating_mul(std::mem::size_of::<Series>());
charge_samples(buffer, &output, output_vec_bytes, queue_bytes)?;
return Ok(Some(AggregateOutput::Samples(output)));
}
Err(no_fold(&self.describe()))
}
}
/// Charges the series a finished aggregate hands over, plus the queue they enter.
///
/// The queue's own backing is counted by the caller that appends to it, so what is added here is
/// the series themselves and the vector that carries them; the two are charged together because
/// the vector outlives the call that built it.
fn charge_samples(
buffer: &mut NodeBuffer,
output: &[Series],
output_vec_bytes: usize,
queue_bytes: usize,
) -> Result<()> {
let output_bytes = output.iter().fold(0usize, |bytes, series| {
bytes.saturating_add(series_bytes(series))
});
buffer.resize(
output_bytes
.saturating_add(output_vec_bytes)
.saturating_add(queue_bytes),
)
}
/// The rejection of a chunk that is not the layout this node folds.
///
/// The fold indexes its states by absolute grid position and by field, so a chunk of another field
@@ -162,6 +162,14 @@ fn aggregate_over(inner: NativeNode, out: Arc<EvalGrid>, stage: AggregateStage)
AggregateStage::Partial => {
NativeType::AggregateState(Arc::new(crate::native::plan::AggregateStateSchema {
group: Grouping::By(Arc::from([])),
// The state family this branch carries: the kind and its components come from the
// kernel binding, not from a second mapping.
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from(
crate::native::plan::aggregate_state_binding(AggregateKind::Sum)
.expect("sum has a mergeable state")
.1,
),
accumulators: fields(1),
grid: Arc::clone(&out),
}))
+249 -47
View File
@@ -53,7 +53,7 @@ use crate::map::{
map_compare_series, map_math, map_math_with_parameters,
};
use crate::map_time::{self, DateFunction};
use crate::native::aggregate::AggregateRuntime;
use crate::native::aggregate::{AggregateOutput, AggregateRuntime};
use crate::native::batch::EvaluatedChunk;
use crate::native::metrics::NodeMetrics;
use crate::native::plan::{
@@ -63,6 +63,7 @@ use crate::native::plan::{
#[cfg(test)]
pub(crate) use crate::native::runs::absorb_rebuilds;
use crate::native::runs::{Run, absorb};
use crate::native::state::{StateChunk, state_bytes};
use crate::range_eval::RangeEvaluator;
use crate::select::{self, SortOrder};
use crate::vector_match::{
@@ -78,12 +79,13 @@ mod tests;
/// Only a source hands over raw samples; everything above a window hands over evaluated chunks.
/// The two cannot be confused once the tree is built, because a window's bound input grid says
/// which one its input produces.
#[derive(Clone, Debug)]
enum NativeChunk {
/// Raw source samples, with their own timestamps.
Raw(RawSeriesChunk),
/// Evaluated values, addressed on their absolute grid positions.
Evaluated(EvaluatedChunk),
/// Partial aggregate state, addressed on its group's absolute grid positions.
AggregateState(StateChunk),
}
impl NativeChunk {
@@ -92,6 +94,19 @@ impl NativeChunk {
match self {
Self::Raw(chunk) => chunk.time.len(),
Self::Evaluated(chunk) => chunk.series.time.len(),
Self::AggregateState(StateChunk::Series { series, .. }) => {
series.iter().map(|s| s.time.len()).sum()
}
Self::AggregateState(StateChunk::Row(row)) => row.time.len(),
}
}
/// What this chunk is, for a rejection that has to say which kind arrived.
fn describe(&self) -> &'static str {
match self {
Self::Raw(_) => "raw samples",
Self::Evaluated(_) => "an evaluated chunk",
Self::AggregateState(_) => "aggregate state",
}
}
@@ -103,6 +118,11 @@ impl NativeChunk {
match self {
Self::Raw(chunk) => chunk.is_last,
Self::Evaluated(chunk) => chunk.is_last,
// A state chunk is what a partition's fold produces after its input ended, so the
// groups it names are closed by the time they travel. A `Final` closes a group on the
// end of its whole input rather than on one chunk, so nothing reads this as "the group
// is complete"; it only says the producer is done with these groups.
Self::AggregateState(_) => true,
}
}
@@ -116,6 +136,7 @@ impl NativeChunk {
match self {
Self::Raw(chunk) => raw_bytes(chunk),
Self::Evaluated(chunk) => evaluated_bytes(chunk),
Self::AggregateState(chunk) => state_bytes(chunk),
}
}
}
@@ -477,6 +498,26 @@ impl NodeBuffer {
Ok(())
}
/// Moves `bytes` of what this node holds into a reservation of its own.
///
/// The bytes stay charged to the pool: only who holds them changes, so a consumer that keeps
/// the handed-over chunk remains accountable for it. The returned reservation reports what it
/// took, which is what the caller no longer holds.
fn transfer(&mut self, bytes: usize) -> Result<MemoryReservation> {
// A chunk is queued with the figure this node grew by, so the two agree unless the
// accounting drifted. Refusing is what keeps a drift from either panicking inside the
// memory path or handing out a chunk nobody is charged for.
if bytes > self.reservation.size() {
return Err(DataFusionError::Internal(format!(
"a queued chunk holds {bytes} bytes, more than the {} its node reserves",
self.reservation.size()
)));
}
let reservation = self.reservation.split(bytes);
self.record();
Ok(reservation)
}
/// Reports what this node now reserves.
fn record(&self) {
self.metrics.record_reserved_bytes(self.reservation.size());
@@ -2504,10 +2545,13 @@ fn window_workspace_bytes(
}
/// The time of the last sample a chunk carries, on its own timeline.
fn last_sample_ms(chunk: &NativeChunk) -> Option<i64> {
fn last_sample_ms(chunk: &NativeChunk) -> Result<Option<i64>> {
match chunk {
NativeChunk::Raw(chunk) => chunk.time.as_buffer().last().copied(),
NativeChunk::Evaluated(chunk) => last_series_ms(&chunk.series),
NativeChunk::Raw(chunk) => Ok(chunk.time.as_buffer().last().copied()),
NativeChunk::Evaluated(chunk) => Ok(last_series_ms(&chunk.series)),
NativeChunk::AggregateState(_) => Err(DataFusionError::Execution(
"an aggregate state chunk reached a node that reads values or samples".to_string(),
)),
}
}
@@ -2648,10 +2692,56 @@ struct NodeState {
buffer: NodeBuffer,
}
/// The kind of chunk an input leaf is bound to.
///
/// The binding is the tree's own statement about what its input produces, so it — and not a
/// property read off a schema — decides what may be queued here. Raw and evaluated inputs carry a
/// series schema; a state input carries the aggregate-state binding instead, and neither may be
/// fed the other's chunks.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum LeafKind {
Raw,
Evaluated,
AggregateState,
}
impl LeafKind {
/// What a leaf bound to `output` accepts.
fn of(output: &NativeType) -> Result<Self> {
match output {
NativeType::Samples(schema) if schema.eval_grid.is_some() => Ok(Self::Evaluated),
NativeType::Samples(_) => Ok(Self::Raw),
NativeType::AggregateState(_) => Ok(Self::AggregateState),
NativeType::Scalar(_) => Err(DataFusionError::Execution(
"an input leaf cannot be bound to a scalar".to_string(),
)),
}
}
/// What this kind is called in a rejection.
fn describe(self) -> &'static str {
match self {
Self::Raw => "raw samples",
Self::Evaluated => "an evaluated series",
Self::AggregateState => "aggregate state",
}
}
/// Whether a chunk of `chunk` may be queued at a leaf of this kind.
fn accepts(self, chunk: &NativeChunk) -> bool {
matches!(
(self, chunk),
(Self::Raw, NativeChunk::Raw(_))
| (Self::Evaluated, NativeChunk::Evaluated(_))
| (Self::AggregateState, NativeChunk::AggregateState(_))
)
}
}
/// The caller-owned state of one input leaf.
struct InputSlot {
source_index: usize,
schema: Arc<SeriesSchema>,
kind: LeafKind,
open_source_series: bool,
state: NodeState,
}
@@ -2705,13 +2795,10 @@ impl RuntimeNode {
bound.ordering
)));
}
let schema = node.output.samples().cloned().ok_or_else(|| {
DataFusionError::Execution("input node does not produce samples".to_string())
})?;
let id = InputId(inputs.len());
inputs.push(InputSlot {
source_index: bound.source_index,
schema,
kind: LeafKind::of(&node.output)?,
open_source_series: false,
state: NodeState::new(
format!(
@@ -3572,6 +3659,12 @@ impl RuntimeNode {
.to_string(),
));
}
Some(NativeChunk::AggregateState(_)) => {
return Err(DataFusionError::Execution(
"an aggregate state chunk reached a node that reads values or samples"
.to_string(),
));
}
None => return Ok(None),
};
if let Some(output) = state
@@ -4199,11 +4292,10 @@ impl NativeExecutionState {
"native runtime root produces raw samples".to_string(),
));
}
NativeType::AggregateState(_) => {
return Err(DataFusionError::Execution(
"native runtime root produces no sample output".to_string(),
));
}
// A tree rooted at a `Partial` aggregate answers state, which the caller pulls as
// state. It is still an answer, so nothing is refused here; what the root produces is
// read by `NativeExecutor::next_state` rather than `next_chunk`.
NativeType::AggregateState(_) => {}
}
Ok(Self {
root,
@@ -4372,16 +4464,31 @@ impl NodeState {
/// input's figure has always been its positions, while a kernel's result holds one value per
/// field and a field may have none at a position it decided.
fn pop_queued(&mut self, values: fn(&NativeChunk) -> usize) -> Result<Option<NativeChunk>> {
let Some((chunk, bytes)) = self.queued.pop_front() else {
let Some((mut chunk, bytes)) = self.queued.pop_front() else {
return Ok(None);
};
// A producer hands its state to a consumer that packs rows from the batch across polls, so
// that chunk's charge travels with it instead of being returned here.
let moved = match &mut chunk {
NativeChunk::AggregateState(StateChunk::Series { reservation, .. }) => {
debug_assert!(
reservation.is_none(),
"a handed-over state chunk was charged to a queue twice"
);
let transferred = self.buffer.transfer(bytes)?;
let moved = transferred.size();
*reservation = Some(transferred);
moved
}
_ => 0,
};
let metrics = Arc::clone(&self.buffer.metrics);
{
// An input node's own work is handing the chunk over, so its self time covers that and
// nothing its producer did. The chunk leaves this node, so the pool no longer has to
// hold it.
// nothing its producer did. Whatever the chunk did not take with it, the pool no longer
// has to hold.
let _timer = metrics.timer();
self.buffer.shrink(bytes)?;
self.buffer.shrink(bytes - moved)?;
self.release_queued_tables()?;
}
metrics.record_output(1, values(&chunk));
@@ -4702,10 +4809,16 @@ impl NodeState {
let (fields, metric) = match &chunk {
NativeChunk::Raw(chunk) => (&chunk.fields, chunk.metric.as_ref()),
NativeChunk::Evaluated(chunk) => (&chunk.series.fields, Some(&chunk.series.metric)),
NativeChunk::AggregateState(_) => {
return Err(DataFusionError::Execution(
"an aggregate state chunk reached a node that reads values or samples"
.to_string(),
));
}
};
let workspace = match metric {
Some(metric) => {
window_workspace_bytes(window, last_sample_ms(&chunk), closing, fields, metric)?
window_workspace_bytes(window, last_sample_ms(&chunk)?, closing, fields, metric)?
}
// The evaluator refuses a raw chunk without labels before it writes a row.
None => 0,
@@ -4724,6 +4837,12 @@ impl NodeState {
// before its input series has ended; buffering the chunks into a whole series first
// would hold the input's bytes and delay every result to the end of the series.
NativeChunk::Evaluated(chunk) => window.push_series(&chunk.series, chunk.is_last),
NativeChunk::AggregateState(_) => {
return Err(DataFusionError::Execution(
"an aggregate state chunk reached a node that reads values or samples"
.to_string(),
));
}
}
};
metrics.record_input(1, positions);
@@ -4907,6 +5026,26 @@ impl NodeState {
aggregate: &mut AggregateRuntime,
chunk: NativeChunk,
) -> Result<()> {
// A `Final` aggregate merges state; the others fold values. Which one a chunk is has
// already been checked against the leaf that queued it, so an unexpected kind here is a
// tree the compiler should have refused.
if let NativeChunk::AggregateState(chunk) = chunk {
let pending = aggregate.pending_state(&chunk);
let positions = match &chunk {
StateChunk::Series { series, .. } => {
series.iter().map(|series| series.time.len()).sum()
}
StateChunk::Row(row) => row.time.len(),
};
self.buffer.grow(pending)?;
let metrics = Arc::clone(&self.buffer.metrics);
{
let _timer = metrics.timer();
aggregate.accumulate_state(chunk, &mut self.buffer, pending)?;
}
metrics.record_input(1, positions);
return Ok(());
}
let NativeChunk::Evaluated(chunk) = chunk else {
return Err(DataFusionError::Execution(
"the aggregate consumes evaluated chunks, not raw samples".to_string(),
@@ -4960,11 +5099,29 @@ impl NodeState {
return Ok(());
};
let mut closed = 0usize;
for series in output {
let bytes = series_bytes(&series);
let chunk = NativeChunk::Evaluated(EvaluatedChunk::try_new(series, true)?);
self.queued.push_back((chunk, bytes));
closed += 1;
match output {
AggregateOutput::Samples(series) => {
for series in series {
let bytes = series_bytes(&series);
let chunk = NativeChunk::Evaluated(EvaluatedChunk::try_new(series, true)?);
self.queued.push_back((chunk, bytes));
closed += 1;
}
}
// A partial aggregate hands its whole partition over as one state chunk: the groups in
// it are the groups this execution folded, and the consumer closes each when its own
// input ends rather than when a chunk arrives.
AggregateOutput::State(chunk) => {
let StateChunk::Series { series, .. } = &chunk else {
return Err(DataFusionError::Internal(
"Partial aggregate produced row state".into(),
));
};
closed = series.len();
let bytes = state_bytes(&chunk);
self.queued
.push_back((NativeChunk::AggregateState(chunk), bytes));
}
}
// A group is one series and this node closes each of them exactly once, on the one chunk it
// hands it over as.
@@ -5296,17 +5453,11 @@ impl InputSlot {
/// no address here, so it is refused rather than queued and read as though it were the kind this
/// leaf promised.
fn enqueue(&mut self, chunk: NativeChunk) -> Result<()> {
let bound = self.schema.eval_grid.is_some();
let evaluated = matches!(chunk, NativeChunk::Evaluated(_));
if evaluated != bound {
if !self.kind.accepts(&chunk) {
return Err(self.state.context(DataFusionError::Execution(format!(
"the native source is bound to {}, but {} chunk was queued at it",
if bound {
"an evaluated series"
} else {
"raw samples"
},
if evaluated { "an evaluated" } else { "a raw" },
"the native source is bound to {}, but {} was queued at it",
self.kind.describe(),
chunk.describe(),
))));
}
self.state
@@ -5417,6 +5568,14 @@ impl NativeExecutor {
self.push_source(input, NativeChunk::Evaluated(chunk))
}
/// Queues one aggregate-state chunk at the input `input` names.
///
/// Only a leaf bound to state accepts one, which is what keeps a state chunk from ever being
/// read as a value: the leaf refuses it before any node above sees it.
pub(crate) fn push_state(&mut self, input: InputId, chunk: StateChunk) -> Result<()> {
self.push_source(input, NativeChunk::AggregateState(chunk))
}
/// The next chunk the root produces, or `None` when nothing has settled yet.
///
/// A window and a map take whole chunks from their input, so nothing beyond the chunk in hand
@@ -5435,6 +5594,56 @@ impl NativeExecutor {
/// that merge itself, so a root below such a window hands over one series per label set; a map
/// at the root hands over pieces, and a caller that needs one series per label set merges them.
pub fn next_chunk(&mut self) -> Result<Option<EvaluatedChunk>> {
match self.next_output()? {
Some(NativeChunk::Evaluated(chunk)) => Ok(Some(chunk)),
Some(NativeChunk::Raw(_)) => {
let error = DataFusionError::Execution(
"the native runtime produces evaluated chunks, but its root produced raw \
samples"
.to_string(),
);
self.stop(&error);
Err(error)
}
Some(NativeChunk::AggregateState(_)) => {
let error = DataFusionError::Execution(
"the native runtime produces evaluated chunks, but its root produced \
aggregate state"
.to_string(),
);
self.stop(&error);
Err(error)
}
None => Ok(None),
}
}
/// The next aggregate state chunk the root produces, or `None` when nothing has settled yet.
///
/// A tree whose root is a `Partial` aggregate answers state rather than values, so a caller
/// that runs one pulls here instead of [`Self::next_chunk`]. The two are separate calls on
/// purpose: a chunk handed up a state tree is not a query value, and accepting one where a
/// value is expected would be a type error the compiler has already ruled out.
pub(crate) fn next_state(&mut self) -> Result<Option<StateChunk>> {
match self.next_output()? {
Some(NativeChunk::AggregateState(chunk)) => Ok(Some(chunk)),
Some(other) => {
let error = DataFusionError::Execution(format!(
"the native runtime produces aggregate state, but its root produced {}",
other.describe()
));
self.stop(&error);
Err(error)
}
None => Ok(None),
}
}
/// The next chunk the root produces, whatever kind it is.
///
/// The kind is the tree's own: its root output decides whether this is a value, a raw chunk or
/// state, so the wrapper methods above narrow rather than guess.
fn next_output(&mut self) -> Result<Option<NativeChunk>> {
self.check_running()?;
// The end of the input is walked here rather than in one call, so a chunk is handed over as
// soon as the tree settles it: the caller pulls the answer instead of receiving it as one
@@ -5443,18 +5652,7 @@ impl NativeExecutor {
// The state's tree is the one this execution was built from, and a pull is asked of its
// root; the nodes below it read their own children from there.
match self.state.pull(self.inputs_finished) {
Ok(Some(NativeChunk::Evaluated(chunk))) => return Ok(Some(chunk)),
// Rejected when the state was built; kept as an error rather than a panic so an
// impossible tree cannot take the process down.
Ok(Some(NativeChunk::Raw(_))) => {
let error = DataFusionError::Execution(
"the native runtime produces evaluated chunks, but its root produced raw \
samples"
.to_string(),
);
self.stop(&error);
return Err(error);
}
Ok(Some(chunk)) => return Ok(Some(chunk)),
// Nothing has settled: the stages below decide whether that is a stall or an end.
Ok(None) => {}
Err(error) => {
@@ -5713,6 +5911,10 @@ fn evaluated_values(chunk: &NativeChunk) -> usize {
match chunk {
NativeChunk::Evaluated(chunk) => valid_values(chunk),
NativeChunk::Raw(chunk) => chunk.time.len(),
NativeChunk::AggregateState(StateChunk::Series { series, .. }) => {
series.iter().map(|series| series.time.len()).sum()
}
NativeChunk::AggregateState(StateChunk::Row(row)) => row.time.len(),
}
}
+503 -58
View File
@@ -686,7 +686,7 @@ fn raw_shared_histogram_chunk(
fn source_runtime(pool: &Arc<dyn MemoryPool>) -> InputSlot {
InputSlot {
source_index: 0,
schema: raw_schema(),
kind: super::LeafKind::Raw,
open_source_series: false,
state: NodeState::new(
"native.input at root (source_index=0)",
@@ -3677,7 +3677,7 @@ fn standalone_merge_keeps_queued_label_bytes_reserved_until_drain() {
const SERIES: usize = 64;
const LABEL_BYTES: usize = 4 * 1024;
let grid = grid(0, 1, 1);
let grid = grid(0, 1, 16);
let tree = NativeTree::try_new(merge(evaluated_input(Arc::clone(&grid)))).unwrap();
let schema = evaluated_schema(grid);
let pool = pool();
@@ -10189,9 +10189,9 @@ fn aggregate_fold_state_is_released_after_input() {
assert_eq!(pool.reserved(), 0);
}
/// An aggregate kind or stage this runtime does not compute is refused, naming what was found.
/// An aggregate kind this runtime does not compute is refused, naming what was found.
#[test]
fn an_aggregate_kind_or_stage_the_runtime_does_not_compute_is_named() {
fn an_aggregate_kind_the_runtime_does_not_compute_is_named() {
let grid = || grid(0, 60_000, 4);
let kind_tree = NativeTree::try_new(aggregate(
evaluated_input(grid()),
@@ -10202,44 +10202,45 @@ fn an_aggregate_kind_or_stage_the_runtime_does_not_compute_is_named() {
},
))
.unwrap();
let partial_tree = {
let out = grid();
let group = Grouping::By(Arc::new(["job".into()]));
let state = Arc::new(AggregateStateSchema {
group: group.clone(),
accumulators: fields(),
grid: Arc::clone(&out),
});
NativeTree::try_new(NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(evaluated_input(Arc::clone(&out))),
op: BoundAggregate {
kind: AggregateKind::Sum,
group,
stage: AggregateStage::Partial,
},
},
output: NativeType::AggregateState(state),
})
.unwrap()
let error = match NativeExecutionState::try_new(&kind_tree, pool(), None) {
Ok(_) => panic!("the runtime accepted a Quantile aggregate"),
Err(error) => error,
};
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"Quantile: {error}"
);
let message = error.to_string();
assert!(message.contains("Quantile"), "Quantile: {message}");
for (found, tree) in [("Quantile", kind_tree), ("Partial", partial_tree)] {
let error = match NativeExecutionState::try_new(&tree, pool(), None) {
Ok(_) => panic!("the runtime accepted a {found} aggregate"),
Err(error) => error,
};
assert!(
matches!(error, DataFusionError::NotImplemented(_)),
"{found}: {error}"
);
let message = error.to_string();
assert!(message.contains(found), "{found}: {message}");
// A `Partial` rejection has to say why it is not computed: the stages it would need.
if found == "Partial" {
assert!(message.contains("Final"), "{message}");
}
}
let out = grid();
let group = Grouping::By(Arc::new(["job".into()]));
let state = Arc::new(AggregateStateSchema {
group: group.clone(),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: fields(),
grid: Arc::clone(&out),
});
let partial_tree = NativeTree::try_new(NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(evaluated_input(Arc::clone(&out))),
op: BoundAggregate {
kind: AggregateKind::Sum,
group,
stage: AggregateStage::Partial,
},
},
output: NativeType::AggregateState(state),
})
.unwrap();
assert!(
NativeExecutionState::try_new(&partial_tree, pool(), None).is_ok(),
"the runtime rejected a Partial aggregate"
);
}
/// An aggregate above a window that collects and merges its input waits for that merge.
@@ -11113,14 +11114,11 @@ fn a_label_change_admits_the_label_it_writes_before_building_it() {
assert_eq!(pool.reserved(), 0);
}
/// An ended execution keeps its tree and its registered counters, and nothing the input was read
/// with: the schema the leaf was bound to and the grid that schema is addressed on go with it.
///
/// The leaf itself stays -- the root still says which routing slot it is, and the handles the caller
/// registered before the run still read what the nodes recorded -- but the slot that fed it is
/// taken, so an executor kept alive past the execution holds no input schema and no materialized
/// grid. Both ways an execution ends are covered: the answer is drained, and a rejected push stops
/// it first.
/// An ended execution keeps its tree and its registered counters, but neither the input schema nor
/// the grid it is addressed on. The leaf itself stays -- the root still says which routing slot it
/// is, and the handles the caller registered before the run still read what the nodes recorded --
/// but the slot that fed it is taken. Both ways an execution ends are covered: the answer is drained,
/// and a rejected push stops it first.
#[test]
fn an_ended_execution_keeps_no_input_schema_or_grid() {
for fails in [false, true] {
@@ -11144,16 +11142,6 @@ fn an_ended_execution_keeps_no_input_schema_or_grid() {
drop(tree);
drop(schema);
drop(out_grid);
// A chunk addresses its grid but holds no schema, so what still holds either of them is the
// slot the input leaf reads through: that is what the terminal checks below ask about.
assert!(
schema_weak.upgrade().is_some(),
"the queued input is not what holds the schema it was bound to"
);
assert!(
grid_weak.upgrade().is_some(),
"the queued input is not what holds the grid it is addressed on"
);
assert!(
root_holds(&execution, holds_input),
"the root is not the input leaf the tree built"
@@ -11188,6 +11176,15 @@ fn an_ended_execution_keeps_no_input_schema_or_grid() {
drop(refused);
}
// The chunks still queued on the execution must not retain either input allocation.
assert!(
schema_weak.upgrade().is_none(),
"queued chunks retained the schema they were built with"
);
assert!(
grid_weak.upgrade().is_none(),
"queued chunks retained the grid they are addressed on"
);
// What the caller registered before the run sees is what the nodes recorded.
assert!(
schema_weak.upgrade().is_none(),
@@ -11422,3 +11419,451 @@ fn point_wise_nodes_reserve_the_series_their_kernel_builds() {
}
}
}
fn aggregate_with_output(input: NativeNode, op: BoundAggregate, output: NativeType) -> NativeNode {
NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(input),
op,
},
output,
}
}
/// A state chunk the producer hands over stays charged until the consumer drops it.
///
/// The producer queues a chunk against its own reservation and gives that reservation back when the
/// chunk leaves the queue. A consumer that keeps the batch - the state exporter packs rows from it
/// across polls - must then be the one holding those bytes, or the pool admits memory nobody
/// accounts for. The charge therefore moves with the chunk, and dropping the chunk returns exactly
/// that figure.
#[test]
fn a_handed_over_state_chunk_keeps_its_charge_until_it_is_dropped() {
let grid = grid(0, 60_000, 2);
let schema = evaluated_schema(Arc::clone(&grid));
let (state_kind, components) = crate::native::plan::aggregate_state_binding(AggregateKind::Sum)
.expect("sum keeps a mergeable state");
let state_schema = Arc::new(AggregateStateSchema {
group: Grouping::By(Arc::new([])),
kind: state_kind,
components: components.into(),
accumulators: schema.fields.clone(),
grid: Arc::clone(&grid),
});
let tree = NativeTree::try_new(NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(evaluated_input(Arc::clone(&grid))),
op: BoundAggregate {
kind: AggregateKind::Sum,
group: Grouping::By(Arc::new([])),
stage: AggregateStage::Partial,
},
},
output: NativeType::AggregateState(Arc::clone(&state_schema)),
})
.unwrap();
let memory_pool = bounded_pool(1 << 20);
let mut exec = executor(&tree, Arc::clone(&memory_pool));
exec.push_evaluated(
InputId::new(0),
evaluated_chunk(&schema, 0, vec![1.0, 2.0], true),
)
.unwrap();
exec.end_of_input().unwrap();
let chunk = exec
.next_state()
.unwrap()
.expect("a partial sum exports one state chunk");
let crate::native::state::StateChunk::Series {
reservation,
state_schema: exported,
..
} = &chunk
else {
panic!("a partial producer exports series state, not a decoded row");
};
assert_eq!(
state_kind, exported.kind,
"the exported state keeps its kind"
);
assert_eq!(
state_schema.components.as_ref(),
exported.components.as_ref(),
"the exported state keeps its components"
);
let carried = reservation
.as_ref()
.expect("the handed-over chunk carries the charge the queue released")
.size();
assert!(carried > 0, "a state chunk is worth more than no bytes");
assert_eq!(
crate::native::state::state_bytes(&chunk),
carried,
"the chunk carries exactly what the same chunk is worth, so nothing is left behind"
);
assert!(
memory_pool.reserved() >= carried,
"the pool still holds the handed-over bytes: reserved {}, carried {carried}",
memory_pool.reserved()
);
let held = memory_pool.reserved();
drop(chunk);
assert_eq!(
held - carried,
memory_pool.reserved(),
"dropping the chunk returns exactly the bytes it carried"
);
drop(exec);
assert_eq!(0, memory_pool.reserved(), "the execution released its own");
}
#[test]
fn partial_final_matches_complete_over_concatenated_input() {
for kind in [
AggregateKind::Sum,
AggregateKind::Count,
AggregateKind::Group,
] {
let grid = grid(0, 60_000, 2);
let schema = evaluated_schema(Arc::clone(&grid));
let (state_kind, components) = crate::native::plan::aggregate_state_binding(kind).unwrap();
let state_schema = Arc::new(AggregateStateSchema {
group: Grouping::By(Arc::new([])),
kind: state_kind,
components: components.into(),
accumulators: schema.fields.clone(),
grid: Arc::clone(&grid),
});
let aggregate_tree = |stage| {
let output = if stage == AggregateStage::Partial {
NativeType::AggregateState(Arc::clone(&state_schema))
} else {
evaluated(Arc::clone(&grid))
};
let input = if stage == AggregateStage::Final {
NativeNode {
expr: NativeExpr::Input(BoundInput {
output: NativeType::AggregateState(Arc::clone(&state_schema)),
source_index: 0,
ordering: SeriesOrdering::Contiguous,
}),
output: NativeType::AggregateState(Arc::clone(&state_schema)),
}
} else {
evaluated_input(Arc::clone(&grid))
};
NativeTree::try_new(aggregate_with_output(
input,
BoundAggregate {
kind,
group: Grouping::By(Arc::new([])),
stage,
},
output,
))
.unwrap()
};
let mut partial_states = Vec::new();
let mut backing_probe = None;
for vals in [vec![1.0, 2.0], vec![3.0, 4.0]] {
let tree = aggregate_tree(AggregateStage::Partial);
let mut exec = executor(&tree, pool());
exec.push_evaluated(InputId::new(0), evaluated_chunk(&schema, 0, vals, true))
.unwrap();
exec.end_of_input().unwrap();
let mut series = Vec::new();
while let Some(chunk) = exec.next_state().unwrap() {
let crate::native::state::StateChunk::Series {
series: chunk_series,
..
} = chunk
else {
panic!("Partial executor returned row state");
};
series.extend(chunk_series.into_iter().map(|series| (series, true)));
}
let source = Box::pin(futures::stream::iter(series.into_iter().map(Ok)));
let frame_state_schema = Arc::new(
datatypes::series::aggregate_state::AggregateStateSchema::try_new(
state_kind,
state_schema.components.to_vec(),
)
.unwrap(),
);
let frames = common_query::series_frame::pack_aggregate_state(
source,
frame_state_schema.clone(),
Arc::clone(&schema),
)
.unwrap();
let decoded = common_query::series_frame::unpack_aggregate_state(
frames,
Some(frame_state_schema),
Some(Arc::clone(&schema)),
)
.unwrap();
use futures::StreamExt;
for row in futures::executor::block_on(decoded.collect::<Vec<_>>()) {
let row = row.unwrap();
let row =
crate::native::state::state_row_from_frame_row(row, &state_schema).unwrap();
if kind == AggregateKind::Sum && backing_probe.is_none() {
let mut row = row.clone();
let field = row.fields.field(0).unwrap();
let value_len = field
.column(0)
.as_any()
.downcast_ref::<Float64Array>()
.unwrap()
.len();
let retained_len = 4096;
let values = Float64Array::from(vec![7.0; retained_len]);
let compensations = Float64Array::from(vec![0.0; retained_len]);
let value_slice = values.slice(0, value_len);
let compensation_slice = compensations.slice(0, value_len);
let item_fields = field.fields().clone();
let sliced = datatypes::arrow::array::StructArray::try_new(
item_fields,
vec![Arc::new(value_slice), Arc::new(compensation_slice)],
None,
)
.unwrap();
let list_field = Arc::new(datatypes::arrow::datatypes::Field::new(
"item",
sliced.data_type().clone(),
false,
));
let list = datatypes::arrow::array::ListArray::new(
list_field.clone(),
datatypes::arrow::buffer::OffsetBuffer::new(
vec![0_i32, value_len as i32].into(),
),
Arc::new(sliced),
None,
);
let fields = datatypes::arrow::array::StructArray::try_new(
vec![Arc::new(datatypes::arrow::datatypes::Field::new(
"v0",
list.data_type().clone(),
false,
))]
.into(),
vec![Arc::new(list)],
None,
)
.unwrap();
row.fields =
datatypes::series::aggregate_state::StateFieldsBatch::try_from_arrow(
fields,
&datatypes::series::aggregate_state::AggregateStateSchema::try_new(
state_kind,
state_schema.components.to_vec(),
)
.unwrap(),
)
.unwrap();
assert_eq!(row.fields.field(0).unwrap().len(), value_len);
assert!(values.values().inner().capacity() > value_len * 100);
assert!(compensations.values().inner().capacity() > value_len * 100);
let retained_value = row
.fields
.field(0)
.unwrap()
.column(0)
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
assert_eq!(
retained_value.values().inner().as_ptr(),
values.values().inner().as_ptr()
);
backing_probe = Some(crate::native::state::StateChunk::Row(row));
}
partial_states.push(crate::native::state::StateChunk::Row(row));
}
}
let tree_final = aggregate_tree(AggregateStage::Final);
if let Some(state) = backing_probe {
let crate::native::state::StateChunk::Row(row) = &state else {
unreachable!("decoded final input is a row");
};
let item = row.fields.field(0).unwrap();
let visible_len = item.len();
let value = item
.column(0)
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
let compensation = item
.column(1)
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
// Each slice shares a primitive backing vector allocated at `retained_len` above.
let backing_bytes = 2 * 4096 * std::mem::size_of::<f64>();
assert_eq!(
visible_len, 2,
"the state row remains the fixture's two points"
);
assert!(value.values().inner().capacity() > visible_len * 100);
assert!(compensation.values().inner().capacity() > visible_len * 100);
// The charge is the row's own extent, not the 4096-element buffers it borrows: a
// Final row is a one-row slice of the frame's list values, and charging the parent per
// row is what multiplied a chained router's figure.
let own = crate::native::state::state_row_bytes(row);
assert!(
own < backing_bytes,
"a row must not be charged the backing it borrows: own {own}, backing {backing_bytes}"
);
assert!(
own >= 2 * visible_len * std::mem::size_of::<f64>(),
"the charge still covers the values the row carries: {own}"
);
let mut constrained = executor(&tree_final, bounded_pool(own - 1));
let error = constrained
.push_state(
InputId::new(0),
crate::native::state::StateChunk::Row(row.clone()),
)
.expect_err("admission must account for the row it queues");
assert!(is_resources_exhausted(&error), "{error}");
assert_eq!(constrained.state().reserved_bytes(), 0);
let sufficient_pool = bounded_pool(own + 64 * 1024);
let mut sufficient = executor(&tree_final, Arc::clone(&sufficient_pool));
sufficient
.push_state(InputId::new(0), state)
.expect("the budget covers the row and its bookkeeping");
assert!(sufficient.state().reserved_bytes() >= own);
assert!(sufficient_pool.reserved() >= own);
drop(sufficient);
assert_eq!(sufficient_pool.reserved(), 0);
}
let mut final_exec = executor(&tree_final, pool());
let mut first_row_cost = None;
for state in partial_states {
// The cost the input queue charges is `state_bytes`, so that is what the assertion
// pins: a row reported as zero bytes would be held unaccounted while it waits.
if first_row_cost.is_none() {
first_row_cost = Some(crate::native::state::state_bytes(&state));
}
final_exec.push_state(InputId::new(0), state).unwrap();
}
assert!(
first_row_cost.unwrap() >= 2 * std::mem::size_of::<f64>(),
"a queued state row must include retained Arrow component buffers"
);
assert!(
final_exec.state().reserved_bytes() > 0,
"the executor must reserve what its input queue holds"
);
final_exec.end_of_input().unwrap();
let mut merged = Vec::new();
while let Some(chunk) = final_exec.next_chunk().unwrap() {
merged.push(chunk);
}
assert_eq!(final_exec.state().reserved_bytes(), 0);
let mut complete = executor(&aggregate_tree(AggregateStage::Complete), pool());
for vals in [vec![1.0, 2.0], vec![3.0, 4.0]] {
complete
.push_evaluated(InputId::new(0), evaluated_chunk(&schema, 0, vals, true))
.unwrap();
}
complete.end_of_input().unwrap();
let mut serial = Vec::new();
while let Some(chunk) = complete.next_chunk().unwrap() {
serial.push(chunk);
}
let (merged, serial) = (groups_of(&merged), groups_of(&serial));
assert_eq!(
merged
.iter()
.map(|g| (&g.0, &g.1, g.2.iter().map(|p| p.0).collect::<Vec<_>>()))
.collect::<Vec<_>>(),
serial
.iter()
.map(|g| (&g.0, &g.1, g.2.iter().map(|p| p.0).collect::<Vec<_>>()))
.collect::<Vec<_>>()
);
// The 1e-10 tolerance follows `partials_merge_to_the_serial_result` in
// `aggregate/tests.rs`; reassociation may change finite sum rounding, so use both
// absolute and relative bounds (with a unit floor for values near zero).
for (actual, expected) in merged
.iter()
.flat_map(|g| &g.2)
.zip(serial.iter().flat_map(|g| &g.2))
{
if kind != AggregateKind::Sum {
assert_eq!(actual.1, expected.1);
} else {
let (a, e) = (actual.1.unwrap(), expected.1.unwrap());
assert!((a - e).abs() <= 1e-10 * e.abs().max(1.0));
}
}
}
}
#[test]
fn exporting_a_partial_charges_the_fold_and_the_groups_it_builds() {
use crate::aggregate::{AggregateOp, FloatAggregator};
let grid = grid(0, 1, 16);
let schema = evaluated_schema(Arc::clone(&grid));
let series = datatypes::series::samples::Series::try_new(
&schema,
metric(),
TimeAxis::Grid(GridRange::try_new(Arc::clone(&grid), 0, 16).unwrap()),
SeriesFields::one(Samples::from_float(Float64Array::from(vec![Some(1.0); 16]))),
)
.unwrap();
let mut aggregator = FloatAggregator::try_new(
&schema,
AggregateOp::Sum,
crate::aggregate::Grouping::By(&[]),
)
.unwrap();
aggregator.update(&series).unwrap();
let fold_bytes = aggregator.state_bytes();
let mut charges = Vec::new();
let partial = aggregator
.finish_partial_budgeted(|bytes| {
charges.push(bytes);
Ok(())
})
.unwrap();
assert_eq!(
1,
charges.len(),
"the export is admitted once, before it runs"
);
// The groups it built, measured as the chunk that carries them is measured.
let (kind, components) =
crate::native::plan::aggregate_state_binding(AggregateKind::Sum).unwrap();
let chunk_bytes =
crate::native::state::state_bytes(&crate::native::state::StateChunk::Series {
series: partial.groups,
state_schema: Arc::new(AggregateStateSchema {
group: Grouping::By(Arc::new([])),
kind,
components: components.into(),
accumulators: schema.fields.clone(),
grid,
}),
reservation: None,
});
assert!(fold_bytes > 0, "the fold holds its blocks");
assert!(chunk_bytes > 0, "the export builds groups");
// The admitted figure covers what the export still reads *and* what it produced, because both
// are live while it runs. Charging only the output would leave the fold's blocks unaccounted.
assert!(
charges[0] >= fold_bytes.saturating_add(chunk_bytes).saturating_sub(2048),
"the admitted figure covers the fold ({fold_bytes}) and the exported groups ({chunk_bytes}): {}",
charges[0]
);
}
+2 -2
View File
@@ -19,8 +19,8 @@
//! counters and gauges, not the nodes that recorded them.
//!
//! The name carries the node's position in the runtime tree: the whole native tree registers into
//! the one [`ExecutionPlanMetricsSet`] the enclosing physical operator (`FusedSeriesExec` or
//! `AggregateFinalizeExec`) owns, and the reader aggregates by name, so two nodes of a kind sharing
//! the one [`ExecutionPlanMetricsSet`] the enclosing physical operator (`FusedSeriesExec`) owns, and
//! the reader aggregates by name, so two nodes of a kind sharing
//! a name would be summed into a single figure. What a consumer can rely on is the grammar
//! `native.<path>.<metric>` with the suffixes of [`NodeMetric::ALL`]: splitting off the final
//! component leaves the metric, and the path before it keys the node within that operator. That key
+2
View File
@@ -35,3 +35,5 @@ pub mod execute;
pub mod metrics;
pub mod plan;
pub(crate) mod runs;
pub(crate) mod split;
pub(crate) mod state;
+80
View File
@@ -58,6 +58,7 @@ impl Drop for NativeNodeDepthGuard {
}
use datafusion::error::{DataFusionError, Result};
use datatypes::series::aggregate_state::{AggregateStateKind, StateComponent};
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use datatypes::series::time::EvalGrid;
use datatypes::series::{SeriesLabels, SeriesOrdering};
@@ -388,12 +389,51 @@ impl NativeType {
pub struct AggregateStateSchema {
/// How the groups its rows are indexed by are identified.
pub group: Grouping,
/// Aggregate whose partial accumulator this state carries.
pub kind: AggregateStateKind,
/// Ordered accumulator components, shared with the kernel layout.
pub components: Arc<[StateComponent]>,
/// The accumulator columns, one per aggregated field.
pub accumulators: Arc<[FieldSpec]>,
/// The grid the state is indexed on.
pub grid: Arc<EvalGrid>,
}
/// Returns the state type implied by an aggregate, if its kernel has a mergeable state.
///
/// The component list comes from the accumulator implementation rather than a second mapping:
/// changing a kernel's state fields therefore changes the native binding in lockstep. A quantile
/// has no partial kernel and cannot be represented by this state family.
pub(crate) fn aggregate_state_binding(
kind: AggregateKind,
) -> Option<(AggregateStateKind, Vec<StateComponent>)> {
use crate::aggregate::{AggregateOp, StateField};
let (state_kind, op) = match kind {
AggregateKind::Sum => (AggregateStateKind::Sum, AggregateOp::Sum),
AggregateKind::Count => (AggregateStateKind::Count, AggregateOp::Count),
AggregateKind::Avg => (AggregateStateKind::Avg, AggregateOp::Avg),
AggregateKind::Min => (AggregateStateKind::Min, AggregateOp::Min),
AggregateKind::Max => (AggregateStateKind::Max, AggregateOp::Max),
AggregateKind::Group => (AggregateStateKind::Group, AggregateOp::Group),
AggregateKind::Stddev => (AggregateStateKind::Stddev, AggregateOp::Stddev),
AggregateKind::Stdvar => (AggregateStateKind::Stdvar, AggregateOp::Stdvar),
AggregateKind::Quantile { .. } => return None,
};
let components = op
.state_fields()
.iter()
.map(|field| match field {
StateField::Value => StateComponent::Value,
StateField::Mean => StateComponent::Mean,
StateField::Compensation => StateComponent::Compensation,
StateField::Count => StateComponent::Count,
StateField::IncrementalMean => StateComponent::IncrementalMean,
})
.collect();
Some((state_kind, components))
}
/// A scan or exchange boundary bound as a node input.
///
/// The binding records what the input was created for: the output a consumer type-checks against,
@@ -1220,6 +1260,26 @@ impl NativeNode {
input_schema.fields.len(),
"a partial aggregate",
)?;
let Some((expected_kind, expected_components)) =
aggregate_state_binding(op.kind)
else {
return Err(DataFusionError::NotImplemented(format!(
"aggregate state is not defined for {:?}",
op.kind
)));
};
if state.kind != expected_kind
|| state.components.as_ref() != expected_components
{
return Err(DataFusionError::NotImplemented(format!(
"partial {:?} aggregate requires state kind {:?} with components {:?}, found {:?} with {:?}",
op.kind,
expected_kind,
expected_components,
state.kind,
state.components
)));
}
if state.grid.spec() != input_grid.spec() {
return Err(invalid(
"aggregate state is indexed on a grid the input does not produce",
@@ -1241,6 +1301,26 @@ impl NativeNode {
input.output.describe()
)));
};
let Some((expected_kind, expected_components)) =
aggregate_state_binding(op.kind)
else {
return Err(DataFusionError::NotImplemented(format!(
"aggregate state is not defined for {:?}",
op.kind
)));
};
if state.kind != expected_kind
|| state.components.as_ref() != expected_components
{
return Err(DataFusionError::NotImplemented(format!(
"final {:?} aggregate requires state kind {:?} with components {:?}, found {:?} with {:?}",
op.kind,
expected_kind,
expected_components,
state.kind,
state.components
)));
}
if state.group != op.group {
return Err(invalid(
"aggregate state groups differ from the binding: state built for \
+34
View File
@@ -16,6 +16,7 @@ use std::sync::Arc;
use datafusion::common::{DFSchema, DataFusionError};
use datatypes::series::SeriesOrdering;
use datatypes::series::aggregate_state::{AggregateStateKind, StateComponent};
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use datatypes::series::time::{EvalGrid, GridSpec};
@@ -151,6 +152,11 @@ fn partial_aggregate(inner: NativeNode) -> NativeNode {
},
output: NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new(["job".into()])),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: Arc::new([]),
grid,
})),
@@ -1041,6 +1047,11 @@ fn an_aggregate_output_matches_its_stage() {
},
output: NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new(["job".into()])),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: Arc::new([]),
grid: value_grid,
})),
@@ -1114,6 +1125,11 @@ fn a_partial_aggregate_cannot_change_the_accumulator_count() {
},
output: NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new(["job".into()])),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: fields(1),
grid: out,
})),
@@ -1158,6 +1174,11 @@ fn a_partial_aggregate_indexes_its_state_on_the_inputs_grid() {
},
output: NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new(["job".into()])),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: Arc::new([]),
grid: grid(0, 30_000, 8),
})),
@@ -1244,6 +1265,11 @@ fn a_final_aggregate_cannot_change_the_field_count() {
},
output: NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new(["job".into()])),
kind: datatypes::series::aggregate_state::AggregateStateKind::Sum,
components: Arc::from([
datatypes::series::aggregate_state::StateComponent::Value,
datatypes::series::aggregate_state::StateComponent::Compensation,
]),
accumulators: fields(2),
grid: Arc::clone(&out),
})),
@@ -1463,6 +1489,8 @@ fn an_input_can_bind_aggregate_state() {
let state_group_mismatch = NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::Without(Arc::new([Arc::from("job")])),
accumulators: schema_a.accumulators.clone(),
kind: schema_a.kind,
components: schema_a.components.clone(),
grid: schema_a.grid.clone(),
}));
let mismatch = NativeNode {
@@ -1479,11 +1507,15 @@ fn an_input_can_bind_aggregate_state() {
let state = NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new([Arc::from("job")])),
accumulators: fields(2),
kind: AggregateStateKind::Sum,
components: Arc::from([StateComponent::Value, StateComponent::Compensation]),
grid: Arc::clone(&state_grid),
}));
let equivalent_state = NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new([Arc::from("job")])),
accumulators: fields(2),
kind: AggregateStateKind::Sum,
components: Arc::from([StateComponent::Value, StateComponent::Compensation]),
grid: Arc::clone(&state_grid),
}));
let state_input = |bound: NativeType, output: NativeType| NativeNode {
@@ -1501,6 +1533,8 @@ fn an_input_can_bind_aggregate_state() {
);
let reversed_accumulators = NativeType::AggregateState(Arc::new(super::AggregateStateSchema {
group: Grouping::By(Arc::new([Arc::from("job")])),
kind: AggregateStateKind::Sum,
components: Arc::from([StateComponent::Value, StateComponent::Compensation]),
accumulators: Arc::from([
FieldSpec {
name: Arc::from("f1"),
+520
View File
@@ -0,0 +1,520 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Where a native aggregate is split so two planes can run it.
//!
//! A `Complete` aggregate folds every contribution to a group, which is only correct when one
//! execution reads all of them. A raw-series scan hands its partitions to separate executions, so the
//! aggregate is split instead: the per-series prefix of the tree runs below the aggregate in the
//! partitions that read series, that aggregate becomes a `Partial` one exporting the state each
//! partition folded, and everything above it runs over the merged series.
//!
//! The split is structural. The innermost aggregate of the tree is replaced by an input node at its
//! own output - that is where state enters - and the nodes above it become the suffix. What cannot be
//! split correctly is refused with [`DataFusionError::NotImplemented`], which the planner reads as
//! "plan this query the existing way".
use std::sync::Arc;
use datafusion::error::{DataFusionError, Result as DfResult};
use datatypes::series::SeriesOrdering;
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use crate::map::MathOp;
use crate::native::plan::{
AggregateKind, AggregateStage, AggregateStateSchema, BoundAggregate, BoundInput, BoundVectorOp,
Grouping, MapOp, NativeExpr, NativeNode, NativeTree, NativeType, aggregate_state_binding,
};
pub(crate) struct Split {
/// The per-series tree below the cut aggregate, which the producer runs.
prefix: NativeNode,
/// The tree above it, starting at the input the merged values enter at.
suffix: NativeNode,
/// The aggregate that was cut out.
op: BoundAggregate,
/// What that aggregate produces: the layout the exported state is indexed by.
schema: Arc<SeriesSchema>,
/// The suffix reads whole histograms rather than individual groups.
histogram: bool,
/// The suffix rewrites at least one label of the histogram routing key.
label_rewrite: bool,
}
/// Whether every histogram-routing key of this aggregate is necessarily empty.
pub(crate) fn histogram_group_is_single(group: &Grouping) -> bool {
matches!(group, Grouping::By(labels) if labels.iter().all(|label| {
crate::histogram_eval::HISTOGRAM_ROUTING_EXCLUDED_LABELS.contains(&label.as_ref())
}))
}
/// Splits the innermost aggregate out of `node`.
///
/// `Ok(None)` is a subtree holding no aggregate. Point-wise operations can run where their groups
/// are owned. Histogram functions can also run there when the state router co-locates every bucket
/// and every histogram whose output labels collide. A label rewrite may change that key, so it is
/// only admitted alongside a partitioned histogram when it cannot change a label in the cut's
/// routing key. A single-owner histogram needs no such restriction. `topk`, a second aggregate and
/// vector-vector matching still need a whole-vector boundary.
///
/// An instant selection or window above the cut cannot be applied to the merged values under the
/// tree's current binding: the cut output is evaluated, whereas these nodes consume raw samples.
pub(crate) fn split(
node: &NativeNode,
fields: &[FieldSpec],
time_index: &str,
) -> DfResult<Option<Split>> {
match &node.expr {
NativeExpr::Input(_) | NativeExpr::Scalar(_) | NativeExpr::EvalTime { .. } => Ok(None),
NativeExpr::Map { input, op } => {
let Some(mut inner) = split(input, fields, time_index)? else {
return Ok(None);
};
let changes_key = op.ops.iter().any(|op| {
let MapOp::Label(rewrite) = op else {
return false;
};
let destination = rewrite.destination();
!crate::histogram_eval::HISTOGRAM_ROUTING_EXCLUDED_LABELS.contains(&destination)
&& match &inner.op.group {
Grouping::By(labels) => {
labels.iter().any(|label| label.as_ref() == destination)
}
Grouping::Without(labels) => {
!labels.iter().any(|label| label.as_ref() == destination)
}
}
});
if changes_key && inner.histogram && !histogram_group_is_single(&inner.op.group) {
return Err(declined(
"a routing-label rewrite around a partitioned histogram",
));
}
inner.label_rewrite |= changes_key;
inner.suffix = NativeNode {
expr: NativeExpr::Map {
input: Box::new(inner.suffix),
op: op.clone(),
},
output: node.output.clone(),
};
Ok(Some(inner))
}
NativeExpr::Aggregate { input, op } => {
check_cut(op, fields, time_index)?;
match split(input, fields, time_index)? {
// A second aggregate above the cut folds the merged values of every group, which
// no partition owns on its own: it would have to be replanned as its own
// `Partial`/`Final` pair above the first one's result.
Some(_) => Err(declined("an aggregate above a split aggregate")),
None => {
admit_prefix(input)?;
let NativeType::Samples(schema) = &node.output else {
return Err(declined("an aggregate that produces no values"));
};
if schema.eval_grid.is_none() {
return Err(declined("an aggregate that produces no values"));
}
Ok(Some(Split {
prefix: (**input).clone(),
// The merged values enter the suffix as the aggregate's own output: the same
// binding and the same value, which is what the tree validates, at source
// `0` of the suffix's own execution - a source index is local to one
// execution, so the suffix names its one input rather than inheriting the
// prefix's.
suffix: NativeNode {
expr: NativeExpr::Input(BoundInput {
source_index: 0,
output: node.output.clone(),
ordering: SeriesOrdering::Contiguous,
}),
output: node.output.clone(),
},
op: op.clone(),
schema: Arc::clone(schema),
histogram: false,
label_rewrite: false,
}))
}
}
}
NativeExpr::Vector {
input,
op: BoundVectorOp::Histogram(function),
} => {
let Some(mut inner) = split(input, fields, time_index)? else {
return Ok(None);
};
// Two histogram calls may drop different labels at successive levels (in particular,
// a native histogram retains `le`), so the first call's partition key is not enough.
if inner.histogram {
return Err(declined("nested histograms above a split aggregate"));
}
if inner.label_rewrite && !histogram_group_is_single(&inner.op.group) {
return Err(declined(
"a routing-label rewrite around a partitioned histogram",
));
}
inner.histogram = true;
inner.suffix = NativeNode {
expr: NativeExpr::Vector {
input: Box::new(inner.suffix),
op: BoundVectorOp::Histogram(*function),
},
output: node.output.clone(),
};
Ok(Some(inner))
}
// TopK ranks the whole vector, not one histogram or aggregation group at a time.
NativeExpr::Vector {
op: BoundVectorOp::TopK { .. },
..
} => Err(declined("a whole-vector operation above a split aggregate")),
NativeExpr::Merge { .. } => Err(declined("an aggregate split across a canonical merge")),
NativeExpr::Instant { input, .. }
| NativeExpr::Window { input, .. }
| NativeExpr::Timestamp { input }
| NativeExpr::Vector { input, .. }
| NativeExpr::VectorOf { input }
| NativeExpr::ParameterizedMap { input, .. } => match split(input, fields, time_index)? {
Some(_) => Err(declined(
"an aggregate under an operation that reads evaluated values",
)),
None => Ok(None),
},
NativeExpr::Binary {
left,
right,
matching,
} => {
// Each side is split once, and at most one of them may hold the innermost aggregate: two
// would be two cuts, which this node cannot merge.
let (left_split, right_split) = (
split(left, fields, time_index)?,
split(right, fields, time_index)?,
);
// A constant operand is broadcast over whatever the other side answers, so applying the
// same operation to the merged values is the operation the query wrote. Any other sibling
// would have to be matched with those values by label, which nothing below this node
// gathers.
let (mut inner, constant_on_the_left) = match (left_split, right_split) {
(None, None) => return Ok(None),
(Some(inner), None) if matches!(right.expr, NativeExpr::Scalar(_)) => {
(inner, false)
}
(None, Some(inner)) if matches!(left.expr, NativeExpr::Scalar(_)) => (inner, true),
_ => {
return Err(declined(
"an aggregate under a binary operation without a constant on the other side",
));
}
};
// The cut side's own suffix takes the place of the aggregate, on the side the query wrote
// it: a non-commutative operator means what that order says, and the matching and the
// output are the ones the compiler bound.
let (left, right) = if constant_on_the_left {
((**left).clone(), inner.suffix)
} else {
(inner.suffix, (**right).clone())
};
inner.suffix = NativeNode {
expr: NativeExpr::Binary {
left: Box::new(left),
right: Box::new(right),
matching: matching.clone(),
},
output: node.output.clone(),
};
Ok(Some(inner))
}
// A set operation keeps samples by whether their match group is present on the other side,
// which the merged values of one side cannot answer.
NativeExpr::Set { left, right, .. } => {
match (
split(left, fields, time_index)?,
split(right, fields, time_index)?,
) {
(None, None) => Ok(None),
_ => Err(declined("an aggregate under a set operation")),
}
}
// The suffix above a split aggregate reads the merged state alone, so it has no source for
// the info series.
NativeExpr::Info { base, info, .. } => {
match (
split(base, fields, time_index)?,
split(info, fields, time_index)?,
) {
(None, None) => Ok(None),
_ => Err(declined("an aggregate under info")),
}
}
}
}
/// The field names of the raw series the prefix reads, in source order.
///
/// Read off the tree's own raw input node, which is the one authority on what the prefix is
/// compiled against: the root is already evaluated on the statement's grid, so its fields are the
/// statement's value columns rather than the raw ones a `by(...)` clause would name. The prefix is
/// one series at a time and the tree reads exactly one source, so the first raw input answers for
/// all of them.
pub(crate) fn prefix_fields(tree: &NativeTree) -> DfResult<Vec<FieldSpec>> {
fn raw_fields(node: &NativeNode) -> Option<Vec<FieldSpec>> {
match &node.expr {
NativeExpr::Input(input) => input
.output
.samples()
.filter(|schema| schema.eval_grid.is_none())
.map(|schema| schema.fields.to_vec()),
NativeExpr::Scalar(_) | NativeExpr::EvalTime { .. } => None,
NativeExpr::Instant { input, .. }
| NativeExpr::Window { input, .. }
| NativeExpr::Map { input, .. }
| NativeExpr::Merge { input }
| NativeExpr::Timestamp { input }
| NativeExpr::Aggregate { input, .. }
| NativeExpr::Vector { input, .. }
| NativeExpr::VectorOf { input } => raw_fields(input),
NativeExpr::ParameterizedMap {
input, parameters, ..
} => raw_fields(input).or_else(|| parameters.iter().find_map(raw_fields)),
NativeExpr::Binary { left, right, .. }
| NativeExpr::Set { left, right, .. }
| NativeExpr::Info {
base: left,
info: right,
..
} => raw_fields(left).or_else(|| raw_fields(right)),
}
}
raw_fields(tree.root()).ok_or_else(|| declined("a tree that reads no raw series"))
}
/// Refuses an aggregate whose state the split cannot carry, and one grouped by a value field.
///
/// Only the eight fixed-width accumulators are carried: a quantile's state is the group's samples, and
/// a non-`Complete` stage is a state or a merge rather than an answer. A `by(...)` naming a value field
/// or the time index cannot be kept either - the state rows are indexed by field and by grid point,
/// so a group that varies with either is not a group - while a label the input does not have is
/// simply absent, exactly as it is in the existing planner.
pub(crate) fn check_cut(
op: &BoundAggregate,
fields: &[FieldSpec],
time_index: &str,
) -> DfResult<()> {
if op.stage != AggregateStage::Complete {
return Err(declined("an aggregate that is not `Complete`"));
}
if matches!(op.kind, AggregateKind::Quantile { .. }) {
return Err(declined("a `quantile` aggregate"));
}
if let Grouping::By(labels) = &op.group
&& labels.iter().any(|label| {
label.as_ref() == time_index
|| fields
.iter()
.any(|field| field.name.as_ref() == label.as_ref())
})
{
return Err(declined("an aggregate grouped by a value field"));
}
Ok(())
}
/// Checks that the per-series prefix is a tree one partition can evaluate on its own.
///
/// The producer runs one execution per partition and folds exactly the series that partition's scan
/// returned, so the prefix has to answer one series from one series' samples: maps are point-wise, a
/// binary operation with a constant side applies that constant to each series its other side answers,
/// and a window or an instant selection reads the raw input itself. A raw input means no evaluation
/// grid - a tree whose input is already evaluated is a subquery's, which one partition cannot evaluate.
///
/// A `clamp` whose maximum is below its minimum answers no value at all, so the fold would see a group
/// with no sample of that field rather than the query the reader wrote.
pub(crate) fn admit_prefix(root: &NativeNode) -> DfResult<()> {
let mut node = root;
loop {
match &node.expr {
NativeExpr::Map { input, op } => {
for map_op in op.ops.iter() {
if let MapOp::Math(MathOp::Clamp { min, max }) = map_op
&& max < min
{
return Err(declined("a `clamp` whose maximum is below its minimum"));
}
}
node = input;
}
// Only a constant side is broadcast: it reaches each series the other side answers, so
// that side is the one the fold runs. Two constants answer one value per grid point and
// read no series, and two vectors are a label match over the whole vector, which is not
// what either half of the split produces.
NativeExpr::Binary { left, right, .. } => {
node = match (&left.expr, &right.expr) {
(NativeExpr::Scalar(_), NativeExpr::Scalar(_)) => {
return Err(declined("a per-series prefix of two constants"));
}
(NativeExpr::Scalar(_), _) => &**right,
(_, NativeExpr::Scalar(_)) => &**left,
(_, _) => {
return Err(declined(
"a per-series prefix that matches two vectors by label",
));
}
};
}
_ => break,
}
}
let raw = |node: &NativeNode| match &node.expr {
NativeExpr::Input(input) => input
.output
.samples()
.is_some_and(|schema| schema.eval_grid.is_none()),
_ => false,
};
match &node.expr {
NativeExpr::Instant { input, .. } | NativeExpr::Window { input, .. } if raw(input) => {
Ok(())
}
_ => Err(declined(
"a per-series prefix that is not one series at a time",
)),
}
}
pub(crate) fn declined(reason: &str) -> DataFusionError {
DataFusionError::NotImplemented(format!(
"the native aggregate split does not carry {reason}"
))
}
/// A split aggregate as the two trees that run it, and the state they exchange between them.
///
/// The prefix runs where the series are read and ends at a `Partial` aggregate exporting the state
/// each partition folded. The suffix runs where every partition's state has arrived and starts at a
/// `Final` aggregate consuming it, so nothing above the cut sees a partial contribution.
pub struct SplitTrees {
/// The prefix, with the cut aggregate bound over it as `Partial`; it answers aggregate state.
pub partial: NativeTree,
/// The `Final` aggregate over that state, and everything the split moved above the cut.
pub final_tree: NativeTree,
}
impl Split {
/// Binds this split as the two trees that run it.
///
/// `Ok(None)` is an aggregate whose kernel keeps no mergeable state, which cannot cross a
/// partition boundary at all.
pub fn into_trees(self) -> DfResult<Option<SplitTrees>> {
let Self {
prefix,
suffix,
op,
schema,
histogram: _,
label_rewrite: _,
} = self;
let Some((kind, components)) = aggregate_state_binding(op.kind) else {
return Ok(None);
};
let Some(grid) = schema.eval_grid.clone() else {
return Err(DataFusionError::Internal(
"a split aggregate has no evaluation grid".into(),
));
};
let state = Arc::new(AggregateStateSchema {
group: op.group.clone(),
kind,
components: components.into(),
accumulators: schema.fields.clone(),
grid,
});
let partial = NativeTree::try_new(NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(prefix),
op: BoundAggregate {
stage: AggregateStage::Partial,
..op.clone()
},
},
output: NativeType::AggregateState(Arc::clone(&state)),
})?;
let merged = NativeNode {
expr: NativeExpr::Aggregate {
input: Box::new(NativeNode {
expr: NativeExpr::Input(BoundInput {
source_index: 0,
output: NativeType::AggregateState(Arc::clone(&state)),
ordering: SeriesOrdering::Contiguous,
}),
output: NativeType::AggregateState(Arc::clone(&state)),
}),
op: BoundAggregate {
stage: AggregateStage::Final,
..op
},
},
output: NativeType::Samples(Arc::clone(&schema)),
};
let root = enter_suffix(suffix, &merged)?;
Ok(Some(SplitTrees {
partial,
final_tree: NativeTree::try_new(root)?,
}))
}
}
/// Puts `merged` at the cut and returns the suffix above it.
///
/// A suffix is the spine of point-wise maps, constant-sided binaries and at most one histogram
/// operation between the cut and the root, with scalar constants preserved. Other whole-vector
/// operations are refused by `split`.
fn enter_suffix(node: NativeNode, merged: &NativeNode) -> DfResult<NativeNode> {
let NativeNode { expr, output } = node;
let rejected = || {
DataFusionError::Internal("the split suffix holds an operation it does not build".into())
};
let expr = match expr {
NativeExpr::Input(_) => return Ok(merged.clone()),
// A constant carries no state: the suffix keeps it as it is, beside the merged values.
NativeExpr::Scalar(_) => {
return Ok(NativeNode { expr, output });
}
NativeExpr::Map { input, op } => NativeExpr::Map {
input: Box::new(enter_suffix(*input, merged)?),
op,
},
NativeExpr::Vector {
input,
op: op @ BoundVectorOp::Histogram(_),
} => NativeExpr::Vector {
input: Box::new(enter_suffix(*input, merged)?),
op,
},
NativeExpr::Binary {
left,
right,
matching,
} => NativeExpr::Binary {
left: Box::new(enter_suffix(*left, merged)?),
right: Box::new(enter_suffix(*right, merged)?),
matching,
},
_ => return Err(rejected()),
};
Ok(NativeNode { expr, output })
}
+282
View File
@@ -0,0 +1,282 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The aggregate-state chunk a native `Partial`/`Final` aggregate passes between executions.
//!
//! A state chunk is not a query value. It names the group a partition folded and the grid
//! positions that group has contributions at, and it carries the accumulator components its
//! aggregate declared -- nothing else. The conversions here are the only place kernel state
//! ([`PartialAggregate`]) and frame state ([`StateSeries`]) meet, so a hole in one is a hole in the
//! other: a field with no contribution at a carried position stays absent through both directions
//! rather than being filled in with a zero that a later merge would count.
use std::mem::{size_of, size_of_val};
use std::sync::Arc;
use datafusion::arrow::array::Array;
use datafusion::error::{DataFusionError, Result};
use datafusion::execution::memory_pool::MemoryReservation;
use datatypes::series::aggregate_state::{
AggregateStateKind, AggregateStateValues, StateComponent, StateSeries,
};
use datatypes::series::frame::StateFrameRow;
use datatypes::series::time::{GridSpan, TimeAxis};
use crate::aggregate::{AggregateOp, PartialAggregate};
use crate::native::plan::AggregateStateSchema;
/// One partition's aggregate state, either in producer series form or as a decoded consumer row.
#[derive(Debug)]
pub(crate) enum StateChunk {
Series {
series: Vec<StateSeries>,
state_schema: Arc<AggregateStateSchema>,
/// The producer charge this chunk carries once it has left its node's queue.
///
/// A state chunk is charged to the node that folded it until the chunk is handed over. The
/// consumer keeps the batch across polls while it packs rows from it, so the charge travels
/// with those series rather than being returned the moment the producer lets go.
reservation: Option<MemoryReservation>,
},
Row(crate::aggregate::state::StateRow),
}
/// The state kind an aggregate's states are states of.
///
/// Every kind this returns is one the kernel keeps a mergeable partial for; `quantile` and the
/// other non-mergeable aggregates are rejected earlier, when the compiler binds the tree.
pub(crate) fn state_kind(op: AggregateOp) -> AggregateStateKind {
match op {
AggregateOp::Sum => AggregateStateKind::Sum,
AggregateOp::Count => AggregateStateKind::Count,
AggregateOp::Avg => AggregateStateKind::Avg,
AggregateOp::Min => AggregateStateKind::Min,
AggregateOp::Max => AggregateStateKind::Max,
AggregateOp::Group => AggregateStateKind::Group,
AggregateOp::Stddev => AggregateStateKind::Stddev,
AggregateOp::Stdvar => AggregateStateKind::Stdvar,
}
}
/// The accumulator components an aggregate's state carries, in kernel order.
///
/// Derived from [`AggregateOp::state_fields`](crate::aggregate::AggregateOp::state_fields) so the
/// frame layout and the kernel cannot drift apart: adding a component to a kernel adds a column to
/// its state frames in the same change.
fn state_components(op: AggregateOp) -> Vec<StateComponent> {
op.state_fields()
.iter()
.map(|field| match field {
crate::aggregate::StateField::Value => StateComponent::Value,
crate::aggregate::StateField::Mean => StateComponent::Mean,
crate::aggregate::StateField::Compensation => StateComponent::Compensation,
crate::aggregate::StateField::Count => StateComponent::Count,
crate::aggregate::StateField::IncrementalMean => StateComponent::IncrementalMean,
})
.collect()
}
/// Rejects a bound state whose kind or component layout is not what `op` produces.
///
/// Both halves matter. A frame carries its kind so a `sum` state is never merged as an `avg` one,
/// and it carries the component list so a merge reads exactly the columns the producer wrote. Two
/// states of one kind over the same field count and grid are still not interchangeable if their
/// components differ, which is why the comparison is over the list and not over its length.
fn ensure_bound_state(op: AggregateOp, state_schema: &AggregateStateSchema) -> Result<()> {
if state_schema.kind != state_kind(op) {
return Err(DataFusionError::Execution(format!(
"aggregate state of kind {:?} is not {:?} state",
state_schema.kind,
state_kind(op)
)));
}
let expected = state_components(op);
if state_schema.components.as_ref() != expected.as_slice() {
return Err(DataFusionError::Execution(format!(
"aggregate state components {:?} are not what {:?} produces: {:?}",
state_schema.components, op, expected
)));
}
Ok(())
}
/// Hands one partition's kernel state over as the frame state a consumer reads.
///
/// The fold already builds the frame's own representation - field-major states on sparse absolute
/// grid positions - so this moves the groups and checks the binding. It copies nothing: the charge
/// the fold reported for those groups is the charge the chunk keeps.
pub(crate) fn state_chunk_from_partial(
partial: PartialAggregate,
state_schema: Arc<AggregateStateSchema>,
) -> Result<StateChunk> {
ensure_bound_state(partial.op, state_schema.as_ref())?;
Ok(StateChunk::Series {
series: partial.groups,
state_schema,
reservation: None,
})
}
/// Builds the executor's Arrow-backed state row from one decoded frame row.
pub(crate) fn state_row_from_frame_row(
row: StateFrameRow,
binding: &AggregateStateSchema,
) -> Result<crate::aggregate::state::StateRow> {
let fields = row
.fields()
.map_err(|error| DataFusionError::Execution(error.to_string()))?;
if fields.kind() != binding.kind {
return Err(DataFusionError::Execution(format!(
"aggregate state row kind {:?} does not match Final binding {:?}",
fields.kind(),
binding.kind
)));
}
if fields.components() != binding.components.as_ref() {
return Err(DataFusionError::Execution(format!(
"aggregate state row components {:?} do not match Final binding {:?}",
fields.components(),
binding.components
)));
}
Ok(crate::aggregate::state::StateRow {
metric: row.metric().clone(),
time: row.time().clone(),
fields,
kind: binding.kind,
})
}
/// The kernel aggregate a state kind merges into.
pub(crate) fn op_of_state(kind: AggregateStateKind) -> AggregateOp {
match kind {
AggregateStateKind::Sum => AggregateOp::Sum,
AggregateStateKind::Count => AggregateOp::Count,
AggregateStateKind::Avg => AggregateOp::Avg,
AggregateStateKind::Min => AggregateOp::Min,
AggregateStateKind::Max => AggregateOp::Max,
AggregateStateKind::Group => AggregateOp::Group,
AggregateStateKind::Stddev => AggregateOp::Stddev,
AggregateStateKind::Stdvar => AggregateOp::Stdvar,
}
}
/// Transient bytes for the selected state array consumed by a decoded row.
///
/// This estimates the selected extent, not a reservation for the batch's shared parent buffers;
/// a router retains and charges those buffers separately while handing the batch downstream.
pub(crate) fn slice_bytes(array: &dyn Array) -> usize {
array
.to_data()
.get_slice_memory_size()
.unwrap_or_else(|_| array.get_array_memory_size())
}
/// Bytes held by one decoded state row's owned wrappers and retained payloads.
pub(crate) fn state_row_bytes(row: &crate::aggregate::state::StateRow) -> usize {
let fields = row.fields.len();
size_of::<StateChunk>()
.saturating_add(size_of::<crate::aggregate::state::StateRow>())
.saturating_add(fields.saturating_mul(size_of::<datafusion::arrow::array::StructArray>()))
.saturating_add(
(0..fields)
.filter_map(|index| row.fields.field(index))
.map(|field| slice_bytes(field))
.fold(0usize, usize::saturating_add),
)
.saturating_add(
row.metric
.name()
.map_or(0, str::len)
.saturating_add(size_of_val(row.metric.tags()))
.saturating_add(
row.metric
.tags()
.iter()
.map(|tag| tag.name.len().saturating_add(tag.value.len()))
.sum::<usize>(),
),
)
.saturating_add(
row.time
.runs()
.count()
.saturating_mul(size_of::<GridSpan>().saturating_add(size_of::<usize>()))
.saturating_add(if row.time.is_empty() {
0
} else {
size_of::<TimeAxis>()
}),
)
}
/// Bytes a state chunk holds: labels, axis runs, and one sparse state vector per field.
///
/// Like [`crate::grid_series::series_bytes`] this is a work estimate rather than a heap
/// measurement -- it counts vector capacities, not allocator overhead -- which is what the
/// reservation it feeds needs: a figure that never under-counts what the executor holds.
pub(crate) fn state_bytes(chunk: &StateChunk) -> usize {
let StateChunk::Series {
series,
state_schema,
..
} = chunk
else {
let StateChunk::Row(row) = chunk else {
unreachable!("a state chunk is either producer series or a decoded row")
};
return state_row_bytes(row);
};
let series_bytes = series.iter().fold(0usize, |total, series| {
let runs = series.time.runs().count();
let labels = series
.metric
.name()
.map_or(0, str::len)
.saturating_add(size_of_val(series.metric.tags()))
.saturating_add(
series
.metric
.tags()
.iter()
.map(|tag| tag.name.len().saturating_add(tag.value.len()))
.sum::<usize>(),
);
let fields = series
.states
.capacity()
.saturating_mul(size_of::<Vec<Option<AggregateStateValues>>>());
let states = series.states.iter().fold(fields, |bytes, field| {
bytes.saturating_add(
field
.capacity()
.saturating_mul(size_of::<Option<AggregateStateValues>>()),
)
});
total
.saturating_add(size_of::<StateSeries>())
.saturating_add(runs.saturating_mul(size_of::<(usize, usize)>()))
.saturating_add(labels)
.saturating_add(states)
});
size_of::<StateChunk>()
.saturating_add(series.capacity().saturating_mul(size_of::<StateSeries>()))
.saturating_add(series_bytes)
.saturating_add(
state_schema
.components
.len()
.saturating_mul(size_of::<StateComponent>()),
)
}
+49 -30
View File
@@ -37,7 +37,8 @@ use datafusion_optimizer::push_down_filter::PushDownFilter;
use datafusion_optimizer::rewrite_set_comparison::RewriteSetComparison;
use datafusion_optimizer::scalar_subquery_to_join::ScalarSubqueryToJoin;
use datafusion_optimizer::simplify_expressions::SimplifyExpressions;
use promql::extension_plan::{AggregateFinalize, FusedSeries, SeriesDivide};
use promql::extension_plan::{FusedOutputFormat, FusedSeries, SeriesDivide};
use promql::native::plan::{AggregateStage, NativeExpr, NativeType};
use substrait::{DFLogicalSubstraitConvertor, SubstraitPlan};
use table::metadata::TableType;
use table::table::adapter::DfTableProviderAdapter;
@@ -115,9 +116,9 @@ pub struct DistPlannerAnalyzer {
///
/// A region input normally has to be encoded to Substrait before it can leave the frontend, and
/// [`PlanRewriter::should_expand`] cuts the plan at the first node that does not encode. In
/// process, a native fused plan (a `FusedSeries` node, or the `AggregateFinalize` merge over one)
/// travels as a `LogicalPlan` instead and skips that gate; other plans retain the ordinary
/// encoding checks.
/// process, a native fused plan (a `FusedSeries` node, or a `FusedSeries` pair reading a
/// `Partial` producer's state) travels as a `LogicalPlan` instead and skips that gate; other
/// plans retain the ordinary encoding checks.
///
/// Whoever builds the query engine sets this, once, from the transport it wired. It is neither a
/// query-time decision nor a property of a region.
@@ -447,7 +448,8 @@ enum RewriterStatus {
struct PlanRewriter {
/// Whether the whole plan encodes, allowing per-node checks to be skipped.
whole_plan_encodable: bool,
/// In-process root shapes can contain AggregateFinalize, which has no Substrait serializer.
/// In-process root shapes can contain the `Final` half of a split aggregate, which reads every
/// partition's state and so has no Substrait serializer.
skip_in_process_encodability: bool,
/// Whether an in-process root fused plan was recognized for placement-gate evaluation.
in_process_fused_root: bool,
@@ -572,40 +574,57 @@ fn is_fused_series_plan(plan: &LogicalPlan) -> bool {
}
/// Whether `plan` is the native route's whole plan: a `FusedSeries` node over its scan, or the
/// `AggregateFinalize` merge of exactly such a node's partition-local state.
/// `Final` half of a split aggregate over the `Partial` half that reads such a scan.
///
/// The merge is the one shape that puts a node above the fused producer: it runs on the frontend over
/// the state every partition exports, and the producer below it still reads its own scan. Every other
/// plan keeps the ordinary encoding checks.
/// The split is the one shape that puts a node above the fused producer: the `Final` half runs on the
/// frontend over the state every partition exported, and the `Partial` half below it still reads its
/// own scan. Every other plan keeps the ordinary encoding checks.
///
/// The producer is checked by what it exports, not only by its node kind: a `FusedSeries` that
/// answers with computed values is not a merge input, and merging those values as if they were
/// partitions' contributions would report one partition's answer as the whole group's. The check is
/// not implied by the constructor either — `with_exprs_and_inputs` hands a node whatever input an
/// optimizer rebuilt, so a merge can hold a replacement producer its own constructor never saw.
/// The exception applies only when the outer node reads aggregate state, and its inner producer is
/// a `SeriesFrame` over one table scan whose root is a `Partial` aggregate producing
/// `AggregateState`. A `SeriesFrame` can also contain evaluated values, so its format and input
/// shape alone do not establish that it exports state.
///
/// One wrapper may sit above the root: the projection that publishes a statement's own column names
/// over a node whose internal fields are the metric's. It counts only when every expression is a
/// column or an alias of one, and only one such wrapper is allowed - a computed expression or a
/// second projection is an ordinary plan and keeps the encoding checks.
fn is_in_process_plan(plan: &LogicalPlan) -> bool {
if is_fused_series_plan(plan) {
let (root, projected) = match plan {
LogicalPlan::Projection(projection) => (projection.input.as_ref(), Some(projection)),
_ => (plan, None),
};
if let Some(projection) = projected
&& (matches!(projection.input.as_ref(), LogicalPlan::Projection(_))
|| projection.expr.iter().any(|expr| match expr {
Expr::Column(_) => false,
Expr::Alias(alias) => !matches!(alias.expr.as_ref(), Expr::Column(_)),
_ => true,
}))
{
return false;
}
let LogicalPlan::Extension(extension) = root else {
return false;
};
let Some(fused) = extension.node.as_any().downcast_ref::<FusedSeries>() else {
return false;
};
if !fused.reads_aggregate_state() {
// A node reading its own scans is the route's whole plan by itself: one scan, or the
// same-table one-to-one pair the route admits over two.
return true;
}
let LogicalPlan::Extension(extension) = plan else {
let [LogicalPlan::Extension(partial)] = fused.children() else {
return false;
};
if !extension.node.as_any().is::<AggregateFinalize>() {
return false;
}
// The merge's own child, as this rewrite walks it: the producer over its scan.
let inputs = extension.node.inputs();
let [LogicalPlan::Extension(partial)] = inputs.as_slice() else {
let Some(partial) = partial.node.as_any().downcast_ref::<FusedSeries>() else {
return false;
};
let Some(fused) = partial.node.as_any().downcast_ref::<FusedSeries>() else {
return false;
};
fused.partial_fold().is_some()
&& matches!(
partial.node.inputs().as_slice(),
[LogicalPlan::TableScan(_)]
)
partial.format() == FusedOutputFormat::SeriesFrame
&& matches!(partial.children(), [LogicalPlan::TableScan(_)])
&& matches!(&partial.tree().root().expr, NativeExpr::Aggregate { op, .. } if op.stage == AggregateStage::Partial)
&& matches!(&partial.tree().root().output, NativeType::AggregateState(_))
}
impl PlanRewriter {
+81
View File
@@ -632,6 +632,87 @@ fn unencodable_unnest_stays_above_merge_scan() {
}
}
/// The naming projection over a native plan keeps the in-process exception, and nothing wider.
///
/// A split aggregate publishes the statement's own value names from a projection above the `Final`
/// node, so the exception has to see through that one wrapper. It must not see through computed
/// projections, stacked wrappers, or a projection over a shape that was never admitted.
#[test]
fn a_naming_projection_does_not_widen_the_in_process_exception() {
let table = TestTable::table_with_name(0, "t".to_string());
let plan = hybrid_fused(
LogicalPlanBuilder::scan(
"t",
Arc::new(DefaultTableSource::new(Arc::new(
DfTableProviderAdapter::new(table),
))),
None,
)
.unwrap()
.build()
.unwrap(),
"t",
);
assert!(
is_in_process_plan(&plan),
"the bare native root is the exception"
);
// One column-or-alias projection is the naming layer the route builds.
let named = LogicalPlanBuilder::from(plan.clone())
.project(vec![col("v0").alias("sum(t.v0)")])
.unwrap()
.build()
.unwrap();
assert!(
is_in_process_plan(&named),
"a column alias above the native root keeps the exception"
);
// Anything else loses it, because the exception is not a general wrapper allowance.
let computed = LogicalPlanBuilder::from(plan.clone())
.project(vec![(col("v0") + lit(1.0)).alias("sum(t.v0)")])
.unwrap()
.build()
.unwrap();
assert!(
!is_in_process_plan(&computed),
"a computed projection is not a rename"
);
let stacked = LogicalPlanBuilder::from(named)
.project(vec![col("sum(t.v0)")])
.unwrap()
.build()
.unwrap();
assert!(
!is_in_process_plan(&stacked),
"the exception admits one wrapper, not two"
);
// The same projection over a plan that is not the native route keeps the exception off too.
let foreign = LogicalPlanBuilder::from(
LogicalPlanBuilder::scan(
"t",
Arc::new(DefaultTableSource::new(Arc::new(
DfTableProviderAdapter::new(TestTable::table_with_name(0, "t".to_string())),
))),
None,
)
.unwrap()
.build()
.unwrap(),
)
.project(vec![col("number").alias("sum(t.number)")])
.unwrap()
.build()
.unwrap();
assert!(
!is_in_process_plan(&foreign),
"a rename above an ordinary scan is not the exception"
);
}
#[test]
fn dictionary_literals_are_unwrapped_before_pushdown() {
let table = Arc::new(Table::new(
+148 -95
View File
@@ -21,10 +21,10 @@ use datafusion::error::Result as DfResult;
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
use datafusion_expr::{Expr, LogicalPlan, UserDefinedLogicalNode};
use promql::extension_plan::{
AggregateFinalize, EmptyMetric, FusedOutputFormat, FusedSeries, InstantManipulate,
RangeManipulate, RawSeriesSource, SeriesDivide, SeriesNormalize,
EmptyMetric, FusedOutputFormat, FusedSeries, InstantManipulate, RangeManipulate,
RawSeriesSource, SeriesDivide, SeriesNormalize,
};
use promql::native::plan::reads_across_series;
use promql::native::plan::{AggregateStage, NativeExpr, reads_across_series};
use store_api::metric_engine_consts::DATA_SCHEMA_TSID_COLUMN_NAME;
use crate::dist_plan::MergeScanLogicalPlan;
@@ -289,10 +289,19 @@ impl Categorizer {
return Commutativity::NonCommutative;
}
let root = fused.tree().root();
// A `Partial` root is safe below the region boundary only when its prefix itself
// reads per-series work. Do not rely on split admission here: hand-built trees can
// put a whole-vector operation below the aggregate. The `Final` half above the
// exchange is not classified here because its input is the exchange, not a scan.
let partial = matches!(
&root.expr,
NativeExpr::Aggregate { input, op }
if op.stage == AggregateStage::Partial && !reads_across_series(input)
);
let per_series = pointwise_operand(root) || independent_scan_pair(root);
// A gather of one raw scan does not establish region-global ownership. Cross-series
// consumers need their existing split or frontend placement.
if reads_across_series(root) && !per_series {
if reads_across_series(root) && !per_series && !partial {
return Commutativity::NonCommutative;
}
@@ -309,9 +318,6 @@ impl Categorizer {
Commutativity::NonCommutative
}
}
// The merge reads every partition's state, so it cannot run in one of them: a region
// would merge the state its own scan produced and report that as the whole group.
name if name == AggregateFinalize::name() => Commutativity::NonCommutative,
name if name == SeriesNormalize::name()
|| name == InstantManipulate::name()
|| name == RangeManipulate::name() =>
@@ -436,10 +442,9 @@ mod tests {
use datafusion_functions::core::coalesce;
use datafusion_optimizer::analyzer::AnalyzerRule;
use datatypes::series::SeriesOrdering;
use datatypes::series::aggregate_state::{AggregateStateKind, StateComponent};
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use promql::extension_plan::{
AggregateFinalize, FusedSeries, FusedSeriesBinding, RawSeriesSource,
};
use promql::extension_plan::{FusedSeries, FusedSeriesBinding, RawSeriesSource};
use promql::functions::{
NativeHistogramDrop, NativeHistogramFraction, NativeHistogramQuantile,
};
@@ -845,6 +850,8 @@ mod tests {
},
output: NativeType::AggregateState(Arc::new(AggregateStateSchema {
group,
kind: AggregateStateKind::Sum,
components: Arc::from([StateComponent::Value, StateComponent::Compensation]),
accumulators: Arc::from([]),
grid: statement_grid(&native_statement()).unwrap(),
})),
@@ -1031,40 +1038,43 @@ mod tests {
let in_process = analyze(DistPlannerAnalyzer::new(true), plan.clone());
assert!(any_remote_input_holds(&in_process, holds_fused_series));
// A partial producer exports aggregate state; its merge remains frontend-side.
let split = aggregate_finalize_plan(table_scan_plan());
let LogicalPlan::Extension(split_extension) = &split else {
unreachable!()
// A partial producer exports aggregate state, which has no wire form: it serializes only
// inside the region that reads the samples, never across the boundary. Its final half is
// frontend-side for the same reason.
let split = split_aggregate_plan(table_scan_plan());
let final_node = as_final_split(&split).expect("the split's final half");
let [producer] = final_node.children() else {
panic!("expected the final node to have one partial child: {split}")
};
let finalize = split_extension
.node
.as_any()
.downcast_ref::<AggregateFinalize>()
.unwrap();
let inputs = datafusion_expr::UserDefinedLogicalNodeCore::inputs(finalize);
let [producer] = inputs.as_slice() else {
unreachable!()
};
DFLogicalSubstraitConvertor
.encode(*producer, DefaultSerializer)
.expect("partial FusedSeries producer uses its ordinary serializer");
assert!(
DFLogicalSubstraitConvertor
.encode(producer, DefaultSerializer)
.is_err(),
"a state-exporting producer must not cross the wire"
);
let placed = analyze(DistPlannerAnalyzer::new(true), split);
assert!(as_aggregate_finalize(&placed).is_some());
assert!(as_final_split(&placed).is_some());
let child = placed.inputs();
let [child] = child.as_slice() else {
panic!("expected finalize input: {placed}")
panic!("expected the final node's input: {placed}")
};
let scan = first_merge_scan(child).expect("merge scan below finalize");
let scan = first_merge_scan(child).expect("merge scan below the final node");
let producer = first_fused_series(scan.input()).expect("partial producer in region input");
assert!(producer.partial_fold().is_some());
assert!(matches!(
&producer.tree().root().expr,
NativeExpr::Aggregate { op, .. } if op.stage == AggregateStage::Partial
));
assert!(matches!(
&producer.tree().root().output,
NativeType::AggregateState(_)
));
}
/// A split aggregate's merge, over a scan and bound to `sum(some_metric)`.
/// A split aggregate over a scan and bound to `sum(some_metric)`.
///
/// The tree is the compiler's own, so the two halves this plan is built from are the ones a query
/// plans: the aggregate is split out of it, and the producer below the merge runs the per-series
/// prefix.
fn aggregate_finalize_plan(input: LogicalPlan) -> LogicalPlan {
/// The compiler's tree is split into a `Partial` state-exporting node over the scan and a
/// `Final` node above it, matching the two halves a query plans.
fn split_aggregate_plan(input: LogicalPlan) -> LogicalPlan {
let stmt = promql_parser::parser::EvalStmt {
expr: parser::parse("sum(some_metric)").unwrap(),
..native_statement()
@@ -1106,23 +1116,37 @@ mod tests {
.unwrap(),
);
let (partial_tree, final_tree) =
FusedSeries::split_aggregate(&tree, "ts").unwrap().unwrap();
let partial = FusedSeries::try_new_state_export(
Arc::from("sum(some_metric)"),
partial_tree,
binding.clone(),
vec![input],
)
.unwrap();
let final_node = FusedSeries::try_new(
Arc::from("sum(some_metric)"),
final_tree,
binding,
vec![LogicalPlan::Extension(Extension {
node: Arc::new(partial),
})],
output_schema,
)
.unwrap();
LogicalPlan::Extension(Extension {
node: Arc::new(
AggregateFinalize::try_new(
Arc::from("sum(some_metric)"),
tree,
binding,
input,
output_schema,
)
.unwrap(),
),
node: Arc::new(final_node),
})
}
fn as_aggregate_finalize(plan: &LogicalPlan) -> Option<&AggregateFinalize> {
fn as_final_split(plan: &LogicalPlan) -> Option<&FusedSeries> {
if let LogicalPlan::Extension(extension) = plan {
extension.node.as_any().downcast_ref::<AggregateFinalize>()
extension
.node
.as_any()
.downcast_ref::<FusedSeries>()
.filter(|fused| fused.reads_aggregate_state())
} else {
None
}
@@ -1138,82 +1162,111 @@ mod tests {
plan.inputs().into_iter().find_map(first_fused_series)
}
/// The merge reads every partition's state, so it cannot run in one of them.
/// The final half stays above the cut; only the partial half is commutative.
#[test]
fn a_split_aggregate_is_not_commutative() {
let plan = aggregate_finalize_plan(table_scan_plan());
fn a_split_aggregate_is_commutative_only_below_the_cut() {
let plan = split_aggregate_plan(table_scan_plan());
assert!(matches!(
Categorizer::check_plan(&plan, None).unwrap(),
Commutativity::NonCommutative
));
let final_node = as_final_split(&plan).unwrap();
let [partial] = final_node.children() else {
panic!("expected the final node to have one partial child: {plan}");
};
assert!(matches!(
Categorizer::check_plan(partial, None).unwrap(),
Commutativity::Commutative
));
}
/// The merge stays on the frontend while its partial producer rides inside the merge scan.
/// The final half stays on the frontend and the partial half rides inside the region scan.
#[test]
fn a_split_aggregate_merges_above_the_region_with_partial_producer() {
let plan = aggregate_finalize_plan(table_scan_plan());
fn a_split_aggregate_finalizes_above_the_region_only_in_process() {
let plan = split_aggregate_plan(table_scan_plan());
assert!(
DFLogicalSubstraitConvertor
.encode(&plan, DefaultSerializer)
.is_err(),
"the split plan must remain unencodable across the wire"
);
let in_process = analyze(DistPlannerAnalyzer::new(true), plan.clone());
assert!(
as_aggregate_finalize(&in_process).is_some(),
"the merge must stay on the frontend: {in_process}"
as_final_split(&in_process).is_some(),
"the final node must stay on the frontend: {in_process}"
);
let inputs = in_process.inputs();
let [child] = inputs.as_slice() else {
panic!("expected the merge to keep one input in: {in_process}");
let final_node = as_final_split(&in_process).unwrap();
let [child] = final_node.children() else {
panic!("expected the final node to keep one input: {in_process}");
};
// `expand` keeps a projection over the merge scan to restore the schema of what it wrapped,
// so the boundary is the first merge scan under the child.
let merge_scan = first_merge_scan(child)
.unwrap_or_else(|| panic!("expected a merge scan below the merge in: {in_process}"));
let producer = first_fused_series(merge_scan.input()).unwrap_or_else(|| {
panic!("expected the producer in the region input of: {in_process}")
.unwrap_or_else(|| panic!("expected a merge scan below the final node: {in_process}"));
let partial = first_fused_series(merge_scan.input()).unwrap_or_else(|| {
panic!("expected the partial node in the region input: {in_process}")
});
assert!(
producer.partial_fold().is_some(),
"the region input exports state, not values"
);
assert!(
holds_table_scan(merge_scan.input()),
"the producer still reads its own scan in: {in_process}"
);
assert!(!partial.reads_aggregate_state());
assert!(matches!(
&partial.tree().root().expr,
NativeExpr::Aggregate { op, .. } if op.stage == AggregateStage::Partial
));
assert!(holds_table_scan(merge_scan.input()));
// The merge itself has no Substrait serializer, so its frontend placement is unchanged.
let encoded = analyze(DistPlannerAnalyzer::new(false), plan);
assert!(as_aggregate_finalize(&encoded).is_some());
assert!(
!any_remote_input_holds(&encoded, holds_fused_series),
"the analyzer placed a fused node although its plan has to encode: {encoded}"
);
}
/// A merge admits only a producer that exports partial aggregate state. A computed-values
/// producer must be refused by `with_exprs_and_inputs`.
/// The final half of `plan`, with its partial input replaced by `producer`.
fn final_over(plan: LogicalPlan, producer: LogicalPlan) -> LogicalPlan {
let LogicalPlan::Extension(extension) = &plan else {
panic!("a split aggregate is an extension node: {plan}");
};
let fused = extension
.node
.as_any()
.downcast_ref::<FusedSeries>()
.expect("a split aggregate's final half is a FusedSeries");
// Qualified: both `UserDefinedLogicalNode` and `UserDefinedLogicalNodeCore` declare this
// method, and only the latter's answer is the node itself.
let rebuilt =
UserDefinedLogicalNodeCore::with_exprs_and_inputs(fused, Vec::new(), vec![producer])
.expect("the final node holds the one input it is given");
LogicalPlan::Extension(Extension {
node: Arc::new(rebuilt),
})
}
/// The state exception covers the state-exporting producer only. An ordinary values-producing
/// fused child over the same scan is placed by the encoding gate instead: it encodes, so the
/// region may run it, while the final half still never crosses.
#[test]
fn a_merge_requires_a_partial_state_producer_not_computed_values() {
fn an_ordinary_producer_is_placed_by_encodability_not_by_the_state_exception() {
let scan = table_scan_plan();
let split = aggregate_finalize_plan(scan.clone());
// The control: the split's own producer exports state, so it is the shape the exception
// admits and it does ride into the region.
let split = split_aggregate_plan(scan.clone());
let producer_placed = analyze(DistPlannerAnalyzer::new(true), split.clone());
assert!(
any_remote_input_holds(&producer_placed, holds_fused_series),
"the split's partial producer rides inside the merge scan: {producer_placed}"
"the split's partial producer rides inside the final node: {producer_placed}"
);
// The same merge over the same scan, with a producer that answers with values.
let LogicalPlan::Extension(extension) = &split else {
unreachable!()
};
let finalize = extension
.node
.as_any()
.downcast_ref::<AggregateFinalize>()
.unwrap();
let values = fused_series_plan(scan);
let error =
UserDefinedLogicalNodeCore::with_exprs_and_inputs(finalize, Vec::new(), vec![values])
.unwrap_err();
let swapped = final_over(split, values);
let analyzed = analyze(DistPlannerAnalyzer::new(true), swapped);
assert!(
error.to_string().contains("must export aggregate state"),
"{error}"
as_final_split(&analyzed).is_some(),
"the final half never crosses the wire: {analyzed}"
);
assert!(
any_remote_input_holds(&analyzed, holds_fused_series),
"an ordinary fused node encodes, so the region may run it: {analyzed}"
);
assert!(
any_remote_input_holds(&analyzed, holds_table_scan),
"the region still reads the scan the producer reads: {analyzed}"
);
}
+4 -12
View File
@@ -342,10 +342,7 @@ impl PromPlanner {
// a range function and the math functions over it into one tree: offered any later, the
// expression has already been split into the nodes those functions would have shared. A
// shape it does not accept is planned by the existing path exactly as if it were not here.
if let Some(plan) = planner
.try_plan_native_fused(&stmt.expr, query_engine_state)
.await?
{
if let Some(plan) = planner.try_plan_native_fused(&stmt.expr).await? {
return planner.strip_tsid_column(plan);
}
let plan = planner
@@ -1381,10 +1378,8 @@ impl PromPlanner {
self.native_fused && native::hybrid_binary_admitted(*op, modifier, lhs, rhs);
let native = if native_pair {
match (
self.try_plan_native_operand(lhs, query_engine_state)
.await?,
self.try_plan_native_operand(rhs, query_engine_state)
.await?,
self.try_plan_native_operand(lhs).await?,
self.try_plan_native_operand(rhs).await?,
) {
(Some(left), Some(right)) => Some((left, right)),
_ => None,
@@ -2465,10 +2460,7 @@ impl PromPlanner {
ANALYZE_NODE_NAME
| ANALYZE_VERBOSE_NODE_NAME
| EXPLAIN_NODE_NAME
| EXPLAIN_VERBOSE_NODE_NAME => {
self.try_plan_native_fused(&children[0], query_engine_state)
.await?
}
| EXPLAIN_VERBOSE_NODE_NAME => self.try_plan_native_fused(&children[0]).await?,
_ => None,
};
let plan = match native_plan {
+171 -53
View File
@@ -17,9 +17,10 @@
//! [`PromPlanner::try_plan_native_fused`] offers a whole statement expression to the native
//! compiler and, when the compiler accepts it, replaces everything the existing path would plan
//! below the root with one [`FusedSeries`] node over the selector's own table scan. A tree that
//! folds across series - an aggregate - is split instead: its per-series prefix runs in a
//! `FusedSeries` producer and an [`AggregateFinalize`] merges the state every partition exports,
//! because a region holds only part of a group. Declining is the normal answer: every shape the
//! folds across series - an aggregate - is split instead: its per-series prefix and a `Partial`
//! aggregate run in a `FusedSeries` producer, a `SeriesRepartitionExec` routes that state, and a
//! second `FusedSeries` runs the `Final` aggregate over it, because a region holds only part of a
//! group. Declining is the normal answer: every shape the
//! compiler does not accept, and every query this route cannot bind to a scan faithfully, keeps the
//! plan the existing planner builds.
//!
@@ -52,8 +53,8 @@ use datatypes::series::SeriesOrdering;
use datatypes::series::samples::{FieldSpec, SeriesSchema};
use datatypes::series::time::EvalGrid;
use promql::extension_plan::{
AggregateFinalize, FusedOutputFormat, FusedSeries, FusedSeriesBinding, InputLocality,
RawSeriesSource, input_locality, matching_operands_drop_key, tree_source_fields,
FusedOutputFormat, FusedSeries, FusedSeriesBinding, InputLocality, RawSeriesSource,
input_locality, matching_operands_drop_key, tree_source_fields,
};
use promql::native::execute::NativeExecutionState;
use promql::native::plan::{
@@ -76,7 +77,6 @@ use crate::promql::native_compile::{
InfoInput, compile_selectors, compile_statement, info_input, statement_grid,
};
use crate::promql::planner::{DEFAULT_FIELD_COLUMN, DEFAULT_TIME_INDEX_COLUMN, PromPlanner};
use crate::query_engine::QueryEngineState;
/// The query-context key that turns the native fused-series route on for one query.
///
@@ -172,7 +172,6 @@ impl PromPlanner {
pub(super) async fn try_plan_native_operand(
&mut self,
expr: &PromExpr,
_query_engine_state: &QueryEngineState,
) -> Result<Option<(LogicalPlan, super::PromPlannerContext)>> {
let saved = self.ctx.clone();
let (plan, operand_ctx) = match self.try_plan_native_fused_with_context(expr).await? {
@@ -411,7 +410,6 @@ impl PromPlanner {
pub(super) async fn try_plan_native_fused(
&mut self,
prom_expr: &PromExpr,
_query_engine_state: &QueryEngineState,
) -> Result<Option<LogicalPlan>> {
Ok(self
.try_plan_native_fused_with_context(prom_expr)
@@ -739,15 +737,10 @@ impl PromPlanner {
let cross_table =
!same_table_scans(prepared.iter().map(|(scan, ..)| scan)) || logical_ids_differ;
// The first input names the output's time index and, on one table, binds every scan.
let Some((_, series_schema, metric, time, tags, _)) = prepared.first() else {
let Some((_, _, metric, time, tags, _)) = prepared.first() else {
return Ok(None);
};
let (series_schema, metric, time, tags) = (
Arc::clone(series_schema),
metric.clone(),
time.clone(),
tags.clone(),
);
let (metric, time, tags) = (metric.clone(), time.clone(), tags.clone());
// The compiler binds one input per selector in the same pre-order `collect_selectors`
// walks, so position i of both lists is the same occurrence. This is the guard that the
// two walks agree: a mismatch is a compiler defect, not a query this route can plan.
@@ -1019,9 +1012,12 @@ impl PromPlanner {
if multi_input && depends_on_input_order(root) {
return Ok(None);
}
// The native output schema can name only compiled value fields, the time index and tag
// columns. A grouping by metric name has no tag column to represent the distinct groups, so
// this route declines it rather than claiming those groups are interchangeable.
// The native output schema can name only compiled value fields, the time index and the tag
// columns a source declares. A discovered source declares `__name__` because the name is
// what tells two metrics with equal labels apart, so a nameless selector's names survive a
// grouping by `__name__` (#10106). A named selector's source does not declare it - the table
// name is the metric name - so a tree reading more than one such source cannot report the
// distinct groups and declines instead of claiming they are interchangeable.
if multi_input && groups_by_metric_name(root) {
return Ok(None);
}
@@ -1068,48 +1064,39 @@ impl PromPlanner {
// A tree whose fold can be split keeps that plan: it exports aggregate state rather
// than whole samples, so it moves less between the two halves than a sample boundary
// would. The general segmentation below is the fallback for the trees it declines,
// which are the ones its per-series prefix cannot express.
if !multi_input
&& selectors.len() == 1
&& AggregateFinalize::can_split(&tree, &series_schema.fields, &time)
{
// With experimental_native_promql enabled, an aggregate value column is named after
// the compiled tree's root field (for example, `field_0`), not legacy planner text.
// Legacy headers in tests/cases remain unchanged because native PromQL is opt-in/off
// by default.
let Some(output_fields) = native_root_fields(compiled.tree.root()) else {
return Ok(None);
};
let Some(output_tags) = native_output_tags(compiled.tree.root(), &source_tags)
// which are the ones its per-series prefix cannot express. The split is decided before
// anything consumes a child, so a tree it declines keeps the plan it would have built
// without this route.
let split = if !multi_input && selectors.len() == 1 {
FusedSeries::split_aggregate(&tree, &time).context(DataFusionPlanningSnafu)?
} else {
None
};
if let Some((partial, final_tree)) = split {
let Some(output_schema) = Self::fused_root_schema(&final_tree, &source_tags, &time)
else {
return Ok(None);
};
let output_schema = fused_output_schema(&output_fields, &output_tags, &time)?;
let Some(rawinput) = children.into_iter().next() else {
return Ok(None);
};
let node = match AggregateFinalize::try_new(
let node = plan_split_aggregate(
Arc::from(prom_expr.to_string()),
tree.clone(),
partial,
final_tree,
FusedSeriesBinding {
grid,
time_index: time,
time_index: time.clone(),
tags,
metric_name: metric,
},
rawinput,
output_schema,
) {
Ok(node) => node,
Err(DataFusionError::NotImplemented(_)) => return Ok(None),
Err(error) => return Err(error).context(DataFusionPlanningSnafu),
};
return Ok(Some((
LogicalPlan::Extension(Extension {
node: Arc::new(node),
}),
single_input_context,
)));
)?;
let plan = LogicalPlan::Extension(Extension {
node: Arc::new(node),
});
return Ok(Some((plan, single_input_context)));
}
// Preserve the established refusal for repeated same-table operands unless
// segmentation is needed to preserve a computed matching key.
@@ -2242,6 +2229,65 @@ fn fused_output_schema(
))
}
/// Builds the two nodes a split native aggregate runs in.
///
/// The first is a [`FusedSeries`] over `rawinput` running the per-series prefix and a `Partial`
/// aggregate at its root, so each partition exports the state it folded. The second is a
/// [`FusedSeries`] over the first running the `Final` aggregate and everything the split moved
/// above it: it reads state where the first reads series, and answers the statement's rows. Between
/// them the physical planner puts a `SeriesRepartitionExec`, which is what makes every row of a
/// group reach one final partition.
///
/// The statement's output value columns must match the partial state's accumulator count; a
/// mismatch means the declared output shape does not match the state schema of the split tree.
fn plan_split_aggregate(
promql: Arc<str>,
partial: NativeTree,
final_tree: NativeTree,
binding: FusedSeriesBinding,
rawinput: LogicalPlan,
output_schema: DFSchemaRef,
) -> Result<FusedSeries> {
let state_accumulators = match &partial.root().output {
NativeType::AggregateState(state) => state.accumulators.len(),
_ => {
return Err(DataFusionError::Plan(format!(
"the partial tree for `{promql}` does not produce aggregate state"
)))
.context(DataFusionPlanningSnafu);
}
};
let values = output_schema
.fields()
.iter()
.filter(|field| field.data_type() == &ArrowDataType::Float64)
.count();
if values != state_accumulators {
return Err(DataFusionError::Plan(format!(
"the partial state for `{promql}` has {state_accumulators} accumulator(s), but the output schema declares \
{values} value column(s)"
)))
.context(DataFusionPlanningSnafu);
}
let partial = FusedSeries::try_new_state_export(
Arc::clone(&promql),
partial,
binding.clone(),
vec![rawinput],
)
.context(DataFusionPlanningSnafu)?;
FusedSeries::try_new(
promql,
final_tree,
binding,
vec![LogicalPlan::Extension(Extension {
node: Arc::new(partial),
})],
output_schema,
)
.context(DataFusionPlanningSnafu)
}
/// The time-range filter a selector's scan reads, from the bounds its compiled requirement names.
///
/// The selector bounds are already checked against the millisecond timestamp schema during
@@ -2667,9 +2713,9 @@ fn depends_on_input_order(node: &NativeNode) -> bool {
/// than an answer given.
///
/// `without(...)` needs no such check: both routes drop `__name__` there. Nor does a tree with one
/// selector: it reads one metric, so every group it folds carries the same name and naming that name
/// or dropping it cannot merge two groups. What the check is for is the tree that reads more than one
/// metric, which is the one a multi-input node is bound for.
/// source: a discovered selector's source declares `__name__` per metric, and a named one folds one
/// metric whose name cannot merge two groups. What the check is for is the tree reading more than one
/// source, where only a discovered source can report a name at all.
fn groups_by_metric_name(node: &NativeNode) -> bool {
let grouping_names_metric = |op: &BoundAggregate| matches!(&op.group, Grouping::By(names) if names.iter().any(|name| name.as_ref() == METRIC_NAME));
match &node.expr {
@@ -2907,12 +2953,16 @@ mod tests {
use session::context::QueryContextBuilder;
use super::{
METRIC_NAME, NATIVE_PROMQL_EXTENSION_KEY, collect_selectors, cut_segments,
hybrid_binary_admitted, independent_scan_pair, native_output_tags, native_promql_requested,
scan_time_filter,
ArrowDataType, ArrowTimeUnit, METRIC_NAME, NATIVE_PROMQL_EXTENSION_KEY, NativeType,
collect_selectors, cut_segments, fused_output_schema, hybrid_binary_admitted,
independent_scan_pair, native_output_tags, native_promql_requested, scan_time_filter,
};
fn compiled_test_tree(query: &str) -> NativeTree {
compiled_test_tree_with_fields(query, &["v0"])
}
fn compiled_test_tree_with_fields(query: &str, fields: &[&str]) -> NativeTree {
let statement = promql_parser::parser::EvalStmt {
expr: promql_parser::parser::parse(query).unwrap(),
start: UNIX_EPOCH,
@@ -2921,7 +2971,13 @@ mod tests {
lookback_delta: Duration::from_secs(5),
};
let schema = Arc::new(SeriesSchema {
fields: Arc::from([FieldSpec { name: "v0".into() }]),
fields: fields
.iter()
.map(|name| FieldSpec {
name: (*name).into(),
})
.collect::<Vec<_>>()
.into(),
eval_grid: None,
});
crate::promql::native_compile::compile_selectors(
@@ -3336,6 +3392,68 @@ mod tests {
);
}
#[test]
fn numeric_aggregate_projection_keeps_grouping_tags_and_value_schema() {
let sources = [vec![
"host".to_string(),
"kind".to_string(),
"zone".to_string(),
]];
for query in [
"sum by (host, zone) ((x))",
"min by (host, zone) ((x))",
"max without (kind) ((x))",
] {
let tree = compiled_test_tree(query);
let NativeType::Samples(schema) = &tree.root().output else {
panic!("{query} must produce evaluated samples");
};
assert_eq!(1, schema.fields.len(), "{query}");
assert_eq!("v0", schema.fields[0].name.as_ref(), "{query}");
let tags = native_output_tags(tree.root(), &sources).unwrap();
assert_eq!(
vec!["host", "zone"],
tags.iter().map(String::as_str).collect::<Vec<_>>(),
"{query}"
);
}
let tree = compiled_test_tree_with_fields("sum by (zone, host) ((x))", &["v0", "v1"]);
let NativeType::Samples(schema) = &tree.root().output else {
panic!("multi-field aggregate must produce evaluated samples");
};
assert_eq!(
vec!["v0", "v1"],
schema
.fields
.iter()
.map(|field| field.name.as_ref())
.collect::<Vec<_>>()
);
let output = fused_output_schema(
&["v0".to_string(), "v1".to_string()],
&["zone".to_string(), "host".to_string()],
"ts",
)
.unwrap();
assert_eq!(
vec!["ts", "v0", "v1", "zone", "host"],
output
.fields()
.iter()
.map(|field| field.name().as_str())
.collect::<Vec<_>>()
);
assert_eq!(
ArrowDataType::Timestamp(ArrowTimeUnit::Millisecond, None),
*output.field(0).data_type()
);
assert_eq!(ArrowDataType::Float64, *output.field(1).data_type());
assert_eq!(ArrowDataType::Float64, *output.field(2).data_type());
assert_eq!(ArrowDataType::Utf8, *output.field(3).data_type());
assert_eq!(ArrowDataType::Utf8, *output.field(4).data_type());
}
#[test]
fn histogram_quantile_output_tags_drop_bucket_and_metadata_labels() {
let sources = [vec![
+463 -119
View File
@@ -45,9 +45,9 @@ use datatypes::prelude::ConcreteDataType;
use datatypes::schema::{ColumnSchema, Schema};
use datatypes::series::SeriesOrdering;
use promql::extension_plan::{
AggregateFinalize, FusedSeries, HistogramFold, InputLocality, RawSeriesSource, input_locality,
FusedSeries, HistogramFold, InputLocality, RawSeriesSource, input_locality,
};
use promql::native::plan::{NativeExpr, NativeNode};
use promql::native::plan::{AggregateStage, BoundVectorOp, NativeExpr, NativeNode, NativeType};
use promql::vector_match::MatchingLabels;
use promql_parser::label::Labels;
use promql_parser::parser;
@@ -2745,16 +2745,29 @@ async fn native_fused_metric_fields_follow_logical_order() {
}));
}
/// Every raw source nested below `plan` names its own metric occurrence.
/// Every raw source or directly bound scan below `plan` names its own metric occurrence.
fn fused_input_metrics(plan: &LogicalPlan) -> Vec<String> {
fn visit(plan: &LogicalPlan, metrics: &mut Vec<String>) {
if let LogicalPlan::Extension(Extension { node }) = plan {
if let Some(source) = node.as_any().downcast_ref::<RawSeriesSource>() {
metrics.push(source.metric_name().to_string());
} else if let Some(fused) = node.as_any().downcast_ref::<FusedSeries>()
&& let Some(metric) = &fused.binding().metric_name
{
// A single-source segment binds its own metric rather than wrapping its scan in
// RawSeriesSource, even when other segments read different logical tables.
for _ in fused
.children()
.iter()
.filter(|child| matches!(child, LogicalPlan::TableScan(_)))
{
metrics.push(metric.to_string());
}
}
for child in node.inputs() {
visit(child, metrics);
}
}
// Segmented boundaries may add projections above extension nodes.
for child in plan.inputs() {
visit(child, metrics);
}
}
let mut metrics = Vec::new();
@@ -2779,18 +2792,23 @@ async fn native_fused_metric_engine_selectors_keep_their_own_logical_table() {
build_native_fused_plan_with_provider("other_metric", true, provider().await).await;
let fused = find_fused_series(&single).unwrap_or_else(|| panic!("{single}"));
assert_eq!(Some("other_metric"), fused.binding().metric_name.as_deref());
for query in [
"some_metric / other_metric",
"some_metric + other_metric + some_metric",
for (query, expected_metrics) in [
(
"some_metric / other_metric",
vec!["some_metric", "other_metric"],
),
(
"some_metric + other_metric + some_metric",
vec!["some_metric", "other_metric", "some_metric"],
),
] {
let native = build_native_fused_plan_with_provider(query, true, provider().await).await;
assert!(has_pair_node(&native), "{query}: {native}");
let fused = find_fused_series(&native).unwrap_or_else(|| panic!("{query}: {native}"));
assert_eq!(None, fused.binding().metric_name, "{query}");
assert!(
fused_input_metrics(&native)
.iter()
.any(|metric| metric == "other_metric"),
assert_eq!(
expected_metrics,
fused_input_metrics(&native),
"{query}: {native}"
);
}
@@ -2825,18 +2843,37 @@ async fn native_fused_metric_engine_two_aggregate_pair_is_one_execution() {
(DEFAULT_SCHEMA_NAME.to_string(), "other_metric".to_string()),
];
let provider = || build_test_table_provider_with_tsid(&tables, 1, 1);
for query in [
"sum(some_metric) / sum(other_metric)",
"sum(sum(some_metric)) / sum(other_metric)",
"sum by (tag_0) (some_metric) / sum by (tag_0) (other_metric)",
for (query, columns) in [
(
"sum(some_metric) / sum(other_metric)",
vec!["timestamp", "field_0"],
),
(
"sum(sum(some_metric)) / sum(other_metric)",
vec!["timestamp", "field_0"],
),
(
"sum by (tag_0) (some_metric) / sum by (tag_0) (other_metric)",
vec!["timestamp", "field_0", "tag_0"],
),
] {
let native = build_native_fused_plan_with_provider(query, true, provider().await).await;
let fused = find_fused_series(&native)
.unwrap_or_else(|| panic!("{query}: no fused node: {native}"));
assert_eq!(
columns,
native
.schema()
.fields()
.iter()
.map(|field| field.name().as_str())
.collect::<Vec<_>>(),
"{query}: segmented output uses native tree names and order"
);
assert_eq!(2, fused.inputs().len(), "{query}: {native}");
assert!(
fused.partial_fold().is_none(),
"{query}: a multi-input aggregate is not a split: {native}"
find_final_split(&native).is_none(),
"{query}: every input partition folds in one execution: {native}"
);
let input_metrics = fused
.inputs()
@@ -3438,6 +3475,7 @@ async fn native_fused_cross_table_pair_reads_each_table_through_its_own_source()
sources,
"{query}"
);
// This pair route is not an aggregate route: it publishes the node's own field names.
assert_eq!(
vec!["field_0", "tag_0", "timestamp"],
sorted_column_names(&plan),
@@ -3900,10 +3938,24 @@ async fn native_fused_three_selector_trees_bind_every_occurrence() {
fused.binding().metric_name.is_none(),
"{query}"
);
// All four trees take the segmented route, including same-table selectors. The boundary
// publishes the native root's names directly rather than adding a naming projection.
assert!(
matches!(plan, LogicalPlan::Extension(_)),
"{query}: expected the segmented native boundary: {plan}"
);
assert!(
fused.inputs().iter().any(|input| matches!(
input,
LogicalPlan::Extension(Extension { node })
if node.as_any().downcast_ref::<FusedSeries>().is_some()
)),
"{query}: expected a segmented producer: {plan}"
);
assert_eq!(
vec!["field_0", "tag_0", "timestamp"],
sorted_column_names(&plan),
"{query}"
"{query}: the segmented boundary uses the native tree's names"
);
}
}
@@ -3926,37 +3978,27 @@ async fn native_fused_three_selectors_on_partitioned_tables_fall_back() {
}
}
/// The merge half of a split aggregate, when the plan is one.
fn find_aggregate_finalize(plan: &LogicalPlan) -> Option<&AggregateFinalize> {
/// Find the final half of a split aggregate.
fn find_final_split(plan: &LogicalPlan) -> Option<&FusedSeries> {
if let LogicalPlan::Extension(Extension { node }) = plan
&& let Some(finalize) = node.as_any().downcast_ref::<AggregateFinalize>()
&& let Some(fused) = node.as_any().downcast_ref::<FusedSeries>()
&& fused.reads_aggregate_state()
{
return Some(finalize);
return Some(fused);
}
plan.inputs().into_iter().find_map(find_aggregate_finalize)
plan.inputs().into_iter().find_map(find_final_split)
}
/// The producer below a split aggregate, which exports the state one partition folded.
fn partial_producer(finalize: &AggregateFinalize) -> &FusedSeries {
// Bound to a local first: the slice handed out by `inputs` borrows this list.
let inputs = UserDefinedLogicalNodeCore::inputs(finalize);
let [LogicalPlan::Extension(Extension { node })] = inputs.as_slice() else {
panic!("the merge should read one producer: {finalize:?}");
/// The state-exporting producer directly below a final split node.
fn partial_below(final_node: &FusedSeries) -> Option<&FusedSeries> {
let [LogicalPlan::Extension(Extension { node })] = final_node.children() else {
return None;
};
let fused = node
.as_any()
.downcast_ref::<FusedSeries>()
.expect("the producer is a fused series node");
assert!(
fused.partial_fold().is_some(),
"the producer below a merge exports state"
);
fused
node.as_any().downcast_ref::<FusedSeries>()
}
/// The column names of a plan, sorted: the two routes may order the same columns differently, and
/// what has to agree is which columns a result carries.
/// The column names of a plan, sorted for tests that do not depend on physical column order.
fn sorted_column_names(plan: &LogicalPlan) -> Vec<String> {
let mut names = plan
.schema()
@@ -3972,7 +4014,7 @@ fn sorted_column_names(plan: &LogicalPlan) -> Vec<String> {
/// producer. Native names follow the compiled tree; grouping labels, values and timestamps retain
/// their existing schema roles.
#[tokio::test]
async fn native_fused_aggregate_splits_into_a_merge_over_a_partial_producer() {
async fn native_fused_aggregate_splits_into_a_final_over_a_partial_producer() {
for query in [
"sum(some_metric)",
"avg(some_metric)",
@@ -3986,7 +4028,6 @@ async fn native_fused_aggregate_splits_into_a_merge_over_a_partial_producer() {
"sum without (tag_0) (some_metric)",
"sum(rate(some_metric[5m]))",
"ceil(sum(some_metric))",
"sum(avg(some_metric) by (tag_0))",
// A literal scalar on either side of the fold. Below the aggregate it is part of the prefix,
// which is the half that computes the values the state folds; above it the merge rebuilds
// the operation over the merged values, in the order the query wrote the operands. A
@@ -4003,20 +4044,68 @@ async fn native_fused_aggregate_splits_into_a_merge_over_a_partial_producer() {
] {
let plan = build_native_fused_plan(query, true).await;
let display = plan.display_indent().to_string();
assert!(
display.contains("AggregateFinalize: "),
"`{query}`: {display}"
);
assert!(display.contains("[partial "), "`{query}`: {display}");
assert!(find_final_split(&plan).is_some(), "`{query}`: {display}");
assert!(display.contains("TableScan: some_metric"), "{display}");
let finalize = find_aggregate_finalize(&plan).expect("the statement plans to a merge");
let producer = partial_producer(finalize);
// The prefix is the per-series tree; the aggregate itself is not in it.
let final_node = find_final_split(&plan).expect("the statement plans to a final split");
let final_display = final_node.tree().to_string();
assert_eq!(
1,
final_display.matches(" final").count(),
"`{query}`: the merge must be a Final aggregate: {final_display}"
);
let partial = partial_below(final_node).expect("the final node reads a partial producer");
// The physical final must accept the producer's state and classify its own output fields.
let input = Arc::new(datafusion::physical_plan::empty::EmptyExec::new(
Arc::clone(partial.schema().inner()),
));
final_node
.to_execution_plan(vec![(
input,
store_api::region_engine::RawLabelSource::Explicit,
SeriesOrdering::Contiguous,
)])
.unwrap_or_else(|error| panic!("`{query}`: invalid final physical plan: {error}"));
let partial_display = partial.tree().to_string();
let NativeExpr::Aggregate { input, op } = &partial.tree().root().expr else {
panic!("`{query}`: partial root is not an aggregate: {partial_display}");
};
assert_eq!(AggregateStage::Partial, op.stage, "`{query}`");
assert!(
!producer.tree().to_string().contains("Aggregate"),
"`{query}`: {}",
producer.tree()
super::native::pointwise_operand(input),
"`{query}`: the producer prefix must be per-series: {partial_display}"
);
// The cut is the only aggregate in the producer: everything below it is the per-series
// prefix, which folds nothing. A second stage marker would mean the split left an aggregate
// in the half that runs per partition.
assert_eq!(
1,
partial_display.matches(" partial").count(),
"`{query}`: {partial_display}"
);
assert!(
!partial_display.contains(" complete") && !partial_display.contains(" final"),
"`{query}`: {partial_display}"
);
// Final and the statement boundary both publish native field names for physical conversion.
let NativeType::Samples(schema) = &final_node.tree().root().output else {
panic!("`{query}`: final root does not produce samples");
};
for field in schema.fields.iter() {
assert!(
final_node
.schema()
.fields()
.iter()
.any(|column| column.name() == field.name.as_ref()),
"`{query}`: final output is missing native field {}",
field.name
);
}
assert_eq!(
plan.schema(),
final_node.schema(),
"`{query}` native output schema"
);
let expected_names: &[&str] = match query {
"sum(some_metric)" => &["field_0", "timestamp"],
@@ -4049,36 +4138,6 @@ async fn native_fused_aggregate_splits_into_a_merge_over_a_partial_producer() {
}
}
/// Only the innermost aggregate is cut out: everything above it, a map included, runs in the suffix
/// over the merged values.
#[tokio::test]
async fn native_fused_nested_aggregate_splits_at_the_innermost_aggregate() {
for (query, merge, partial) in [
(
"sum(avg(some_metric) by (tag_0))",
r#"AggregateFinalize: sum(avg by (tag_0) (some_metric)) [Avg By(["tag_0"]) Complete]"#,
"[partial avg by (tag_0) Complete]",
),
(
"ceil(sum(rate(some_metric[5m])) by (tag_0))",
r#"AggregateFinalize: ceil(sum by (tag_0) (rate(some_metric[5m]))) [Sum By(["tag_0"]) Complete]"#,
"[partial sum by (tag_0) Complete]",
),
] {
let plan = build_native_fused_plan(query, true).await;
let display = plan.display_indent().to_string();
assert!(display.contains(merge), "{display}");
assert!(display.contains(partial), "{display}");
let finalize = find_aggregate_finalize(&plan).expect("the statement plans to a merge");
let producer = partial_producer(finalize);
// A map above the merge leaves the prefix alone, and so does the outer aggregate.
let prefix = producer.tree().to_string();
assert!(!prefix.contains("Map"), "{query}: {prefix}");
assert!(!prefix.contains("Aggregate"), "{query}: {prefix}");
}
}
/// A data-free vector stays in the gathered parent while the data-bearing prefix is cut.
#[tokio::test]
async fn native_fused_data_free_vector_does_not_block_segmentation() {
@@ -4180,6 +4239,184 @@ async fn native_fused_scalar_broadcast_keeps_segmentation() {
);
}
/// The split boundary takes its ordered names from the Final tree, without an aliasing plan.
#[tokio::test]
async fn native_split_boundary_uses_native_tree_column_names() {
let query = "sum by (tag_0) (abs(some_metric))";
let plan = build_native_fused_plan(query, true).await;
let LogicalPlan::Extension(Extension { node }) = &plan else {
panic!("the Final node must be the output boundary, without a naming projection: {plan}");
};
let final_node = node.as_any().downcast_ref::<FusedSeries>().unwrap();
assert!(final_node.reads_aggregate_state(), "{plan}");
let NativeType::Samples(root) = &final_node.tree().root().output else {
panic!("the Final tree must publish samples: {plan}");
};
let field_names = root
.fields
.iter()
.map(|field| field.name.as_ref())
.collect::<Vec<_>>();
assert_eq!(vec!["field_0"], field_names);
let names = plan
.schema()
.fields()
.iter()
.map(|field| field.name().as_str())
.collect::<Vec<_>>();
assert_eq!(vec!["timestamp", "field_0", "tag_0"], names);
assert_eq!(&names[1..1 + field_names.len()], field_names);
assert!(partial_below(final_node).is_some(), "{plan}");
}
/// Split and complete executions of the same compiled query agree on nonempty values and labels.
#[tokio::test]
async fn native_split_and_complete_aggregate_produce_identical_values() {
use datafusion::arrow::buffer::{Buffer, ScalarBuffer};
use datatypes::series::frame::{FrameLabelMode, pack_raw_series_chunk};
use datatypes::series::raw::{RawSeriesChunk, RawTimestamps};
use datatypes::series::samples::{FieldSpec, Samples, SeriesFields, SeriesSchema};
use datatypes::series::{Label, SeriesLabels};
use store_api::region_engine::RawLabelSource;
let query = "sum by (tag_0) (abs(some_metric))";
let stmt = EvalStmt {
expr: parser::parse(query).unwrap(),
start: UNIX_EPOCH + Duration::from_secs(1),
end: UNIX_EPOCH + Duration::from_secs(1),
interval: Duration::from_secs(1),
lookback_delta: Duration::from_secs(1),
};
let raw_schema = Arc::new(SeriesSchema {
fields: vec![FieldSpec {
name: "field_0".into(),
}]
.into(),
eval_grid: None,
});
let complete_tree = crate::promql::native_compile::compile(
&stmt,
Arc::clone(&raw_schema),
SeriesOrdering::Contiguous,
)
.unwrap();
let provider = build_test_table_provider_with_options(
&[(DEFAULT_SCHEMA_NAME.to_string(), "some_metric".to_string())],
2,
1,
vec![],
ConcreteDataType::timestamp_millisecond_datatype(),
false,
ConcreteDataType::float64_datatype(),
NATIVE_ENGINE,
)
.await;
let state = build_query_engine_state();
let split = PromPlanner::stmt_to_plan_with_annotations(provider, &stmt, &state, None, true)
.await
.unwrap();
let final_node = find_final_split(&split).expect("the query must use a split aggregate");
let partial = partial_below(final_node).unwrap();
let complete = FusedSeries::try_new(
Arc::from(query),
complete_tree,
final_node.binding().clone(),
partial.children().to_vec(),
Arc::clone(final_node.schema()),
)
.unwrap();
let raw = |key: &str, tag: &str, value: f64| {
let labels = SeriesLabels::try_new(
Some("some_metric".into()),
vec![
Label {
name: "tag_0".into(),
value: tag.into(),
},
Label {
name: "tag_1".into(),
value: key.into(),
},
],
)
.unwrap();
let chunk = RawSeriesChunk::try_new(
&raw_schema,
Buffer::from_slice_ref(key.as_bytes()),
Some(labels),
RawTimestamps::try_new(ScalarBuffer::from(vec![1_000])).unwrap(),
SeriesFields::one(Samples::from_float(Float64Array::from(vec![value]))),
true,
)
.unwrap();
pack_raw_series_chunk(&raw_schema, FrameLabelMode::Explicit, None, &chunk).unwrap()
};
// Two partitions contribute to tag a, exercising state merge rather than just partitioning.
let first = [raw("a1", "a", -2.0), raw("b1", "b", 4.0)].concat();
let second = raw("a2", "a", 3.0);
let frame_schema = first[0].schema();
let source = || {
Arc::new(DataSourceExec::new(Arc::new(
MemorySourceConfig::try_new(
&[first.clone(), second.clone()],
Arc::clone(&frame_schema),
None,
)
.unwrap(),
))) as Arc<dyn datafusion::physical_plan::ExecutionPlan>
};
let input = |source| (source, RawLabelSource::Explicit, SeriesOrdering::Contiguous);
let partial_exec = partial.to_execution_plan(vec![input(source())]).unwrap();
let split_exec = final_node
.to_execution_plan(vec![input(partial_exec)])
.unwrap();
let complete_exec = complete.to_execution_plan(vec![input(source())]).unwrap();
let split_batches =
datafusion::physical_plan::collect(split_exec, state.session_state().task_ctx())
.await
.unwrap();
let complete_batches =
datafusion::physical_plan::collect(complete_exec, state.session_state().task_ctx())
.await
.unwrap();
let rows = |batches: &[RecordBatch]| {
let mut rows = batches
.iter()
.flat_map(|batch| {
let tags = batch
.column_by_name("tag_0")
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
let values = batch
.column_by_name("field_0")
.unwrap()
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
let times = batch
.column_by_name("timestamp")
.unwrap()
.as_any()
.downcast_ref::<TimestampMillisecondArray>()
.unwrap();
(0..batch.num_rows())
.map(|i| (tags.value(i).to_string(), times.value(i), values.value(i)))
.collect::<Vec<_>>()
})
.collect::<Vec<_>>();
rows.sort_by(|left, right| left.0.cmp(&right.0));
rows
};
let actual = rows(&split_batches);
assert_eq!(
vec![("a".to_string(), 1_000, 5.0), ("b".to_string(), 1_000, 4.0)],
actual
);
assert_eq!(actual, rows(&complete_batches));
}
/// A nested multi-source matching subtree is built as a segment consuming each source's explicit frames.
#[tokio::test]
async fn native_fused_multi_source_matching_segment_is_constructed() {
@@ -4307,14 +4544,52 @@ async fn native_fused_multi_source_segments_bind_their_own_table_scan() {
);
}
/// A nested aggregate reports the outer operation's native output name.
/// Nested aggregates and their wrappers use the native root's field names, including segmented
/// trees that the aggregate split declines.
#[tokio::test]
async fn native_fused_nested_aggregate_keeps_the_outer_names() {
async fn native_fused_nested_rate_abs_aggregates_use_native_schema_names() {
for query in [
"sum(abs(avg(rate(some_metric[5m])) by (tag_0)))",
"abs(sum(avg(rate(some_metric[5m])) by (tag_0)))",
"ceil(sum(avg(some_metric) by (tag_0)))",
"(sum(avg(some_metric) by (tag_0)) + 1) / 2",
"floor(sum(avg(some_metric) by (tag_0)))",
] {
let native = build_native_fused_plan(query, true).await;
let fused = find_fused_series(&native).unwrap_or_else(|| {
panic!("{query}: native planning must produce a fused-series node: {native}")
});
assert_eq!(
vec!["field_0", "timestamp"],
sorted_column_names(&native),
"{query}: the boundary uses the native tree's field name"
);
assert!(
native
.schema()
.fields()
.iter()
.any(|field| field.data_type() == &ArrowDataType::Float64),
"{query}: native plan must retain a Float64 value field: {native}"
);
let NativeType::Samples(root) = &fused.tree().root().output else {
panic!("{query}: the native root answers samples: {native}");
};
assert_eq!(root.fields[0].name.as_ref(), "field_0", "{query}");
assert_eq!(native.schema(), fused.schema(), "{query}: {native}");
}
}
/// A nested aggregate declines the split but its segmented output still uses the native root name.
#[tokio::test]
async fn native_fused_declined_nested_aggregate_uses_native_name() {
let query = "sum(avg(some_metric) by (tag_0))";
let plan = build_native_fused_plan(query, true).await;
assert!(find_final_split(&plan).is_none(), "{query}: {plan}");
assert_eq!(
sorted_column_names(&plan),
vec!["field_0", "timestamp"],
sorted_column_names(&plan),
"a bare outer sum drops the inner grouping label and names the value after the compiled root field, not legacy expression text"
);
@@ -4334,7 +4609,7 @@ async fn native_fused_clamp_aggregate_is_admitted_without_legacy_registration()
] {
let plan = build_native_fused_plan(query, true).await;
assert!(
find_aggregate_finalize(&plan).is_some() || find_fused_series(&plan).is_some(),
find_fused_series(&plan).is_some(),
"`{query}` should be accepted by native: {}",
plan.display_indent()
);
@@ -4394,10 +4669,9 @@ async fn native_classic_histogram_provider() -> DfTableSourceProvider {
)
}
/// A classic histogram function above the cut is the split's: `histogram_quantile` reads each
/// histogram off the bucket series the merge produced, which are the ones the statement's input
/// would have produced in one execution, so the statement is answered by the merge over the partial
/// producer instead of falling back to the existing plan.
/// Two classic-histogram functions over raw series, each admitted as one native operand: a
/// `histogram_quantile` reads each histogram off the bucket series that share its other labels and
/// drops `le`, so neither operand keeps `le` in its binding nor in the statement's output.
#[tokio::test]
async fn native_fused_histogram_quantile_binary_drops_le_from_operand_matching() {
let provider = native_classic_histogram_provider().await;
@@ -4440,7 +4714,7 @@ async fn native_fused_histogram_quantile_binary_drops_le_from_operand_matching()
};
for operand in [&binary.lhs, &binary.rhs] {
let (_, context) = planner
.try_plan_native_operand(operand, &build_query_engine_state())
.try_plan_native_operand(operand)
.await
.unwrap()
.expect("each histogram call is admitted as one native operand");
@@ -4449,23 +4723,6 @@ async fn native_fused_histogram_quantile_binary_drops_le_from_operand_matching()
}
}
#[tokio::test]
async fn native_fused_histogram_quantile_over_aggregate_splits() {
let query = "histogram_quantile(0.99, sum by (tag_0) (rate(some_metric[5m])))";
let plan = build_native_fused_plan(query, true).await;
let display = plan.display_indent().to_string();
assert!(
find_aggregate_finalize(&plan).is_some(),
"`{query}` must be answered by a split aggregate: {display}"
);
assert!(
find_fused_series(&plan).is_some(),
"`{query}` must keep the producer that exports the state: {display}"
);
assert!(display.contains("AggregateFinalize: "), "{display}");
assert!(display.contains("histogram_quantile"), "{display}");
}
/// A shape the split cannot keep correct keeps the existing plan: a binary operation whose sibling
/// is data would match one side's series before the merge, a sibling a call computes has no native
/// node at all, and a group by a value field cannot be named by the state rows. A literal scalar
@@ -4493,8 +4750,16 @@ async fn native_fused_unsplittable_aggregate_falls_back() {
] {
let native = build_native_fused_plan(query, true).await;
let without_flag = build_native_fused_plan(query, false).await;
assert!(find_fused_series(&native).is_none(), "`{query}`");
assert!(find_aggregate_finalize(&native).is_none(), "`{query}`");
assert!(
find_fused_series(&native).is_none(),
"`{query}`: {}",
native.display_indent()
);
assert!(
find_final_split(&native).is_none(),
"`{query}`: {}",
native.display_indent()
);
assert_eq!(
native.display_indent().to_string(),
without_flag.display_indent().to_string(),
@@ -4503,7 +4768,89 @@ async fn native_fused_unsplittable_aggregate_falls_back() {
}
}
/// Reversed clamp bounds are accepted by the native planner.
/// Suffixes whose answers need the whole vector cannot fuse into a partition-local `Final`, so
/// the split declines them: it neither exports aggregate state nor runs a `Final` aggregate.
/// Ranking (`topk`) and a second aggregate cannot be answered from a partition's groups alone.
/// Classic histogram functions, by contrast, route all buckets of a histogram to the same owner.
///
/// What they fall back to is not asserted: the segmentation route answers some of them, and the
/// point is only that none of them becomes a split.
#[tokio::test]
async fn native_fused_split_declines_a_whole_vector_suffix() {
for (suffix, declined, splits) in [
// Ranks the whole vector: the winner may be in a group this partition does not own.
(
"topk",
"topk(1, sum by (tag_0) (some_metric))",
"sum by (tag_0) (some_metric)",
),
// A second fold over the merged values would have to become its own state split.
(
"a second aggregate",
"sum(avg(some_metric) by (tag_0))",
"avg(some_metric) by (tag_0)",
),
] {
// The control first: the same fold without the suffix is a split, so the refusal below is
// about the suffix and not about a fold this route never takes.
let control = build_native_fused_plan(splits, true).await;
assert!(
find_final_split(&control).is_some(),
"{suffix}: `{splits}` must still split, or the refusal proves nothing: {}",
control.display_indent()
);
let declined_plan = build_native_fused_plan(declined, true).await;
assert!(
find_final_split(&declined_plan).is_none(),
"{suffix}: `{declined}` must not split: {}",
declined_plan.display_indent()
);
}
}
/// A histogram suffix runs after the Final aggregate, with all of each histogram's buckets routed
/// together rather than distributed by `le`.
#[tokio::test]
async fn native_fused_histogram_above_aggregate_splits_with_bucket_routing() {
let query = "histogram_quantile(0.99, sum by (le) (rate(m[5m])))";
let plan = build_native_fused_plan_with_provider(
query,
true,
native_classic_histogram_provider().await,
)
.await;
let display = plan.display_indent().to_string();
let final_node = find_final_split(&plan)
.unwrap_or_else(|| panic!("`{query}` must split into Partial and Final: {display}"));
// Only the Final node over state publishes this suffix, so the plan's own text pins it.
assert!(
display.contains(
" (state routing: canonical labels except [\"le\", \"__name__\", \"__type__\", \"__unit__\"])"
),
"`{query}`: {display}"
);
let NativeExpr::Vector {
input,
op: BoundVectorOp::Histogram(_),
} = &final_node.tree().root().expr
else {
panic!("`{query}`: histogram must run above Final: {display}");
};
assert!(
matches!(&input.expr, NativeExpr::Aggregate { op, .. } if op.stage == AggregateStage::Final),
"`{query}`: histogram must read the Final aggregate: {display}"
);
let partial = partial_below(final_node)
.unwrap_or_else(|| panic!("`{query}`: Final must read a Partial producer: {display}"));
assert!(
matches!(&partial.tree().root().expr, NativeExpr::Aggregate { op, .. } if op.stage == AggregateStage::Partial),
"`{query}`: producer must export Partial state: {display}"
);
}
/// The native route can plan a reversed clamp range without consulting the existing planner,
/// which has no clamp kernel.
#[tokio::test]
async fn native_fused_clamp_reversed_bounds_are_planned_natively() {
let native = build_native_fused_plan("sum(clamp(some_metric, 2, 1))", true).await;
@@ -4553,7 +4900,7 @@ async fn native_fused_clamp_aggregate_is_admitted() {
.await
.unwrap();
assert!(
find_aggregate_finalize(&plan).is_some(),
find_final_split(&plan).is_some(),
"{}",
plan.display_indent()
);
@@ -4649,10 +4996,7 @@ async fn native_fused_aggregate_falls_back_to_legacy_plan() {
let plan = build_native_fused_plan(query, true).await;
let display = plan.display_indent().to_string();
assert!(!display.contains("FusedSeries"), "{query}: {display}");
assert!(
!display.contains("AggregateFinalize:"),
"{query}: {display}"
);
assert!(find_final_split(&plan).is_none(), "{query}: {display}");
assert!(display.contains("Aggregate:"), "{query}: {display}");
let legacy = build_native_fused_plan(query, false).await;
assert_eq!(display, legacy.display_indent().to_string(), "{query}");
File diff suppressed because it is too large Load Diff
+95 -68
View File
@@ -2051,7 +2051,7 @@ async fn native_promql_clamp_aggregates_execute_on_native_segments() {
.to_string();
if split {
assert!(
display.contains("AggregateFinalizeExec"),
display.contains("Aggregate sum by [] final"),
"{query}: {display}"
);
let producer = observer.fused().expect("native partial producer");
@@ -2447,15 +2447,10 @@ async fn native_promql_label_rewrites_reach_the_native_route_and_keep_prometheus
observer.clear();
let without_native = run(without_expr, "(0, 0, '1s')").await.unwrap();
let native_plan = observer.root_plan();
let mut plan_nodes = vec![native_plan];
let mut has_aggregate_finalize = false;
while let Some(node) = plan_nodes.pop() {
has_aggregate_finalize |= node.name() == "AggregateFinalizeExec";
plan_nodes.extend(node.children().into_iter().cloned());
}
assert!(
has_aggregate_finalize,
"native plan must execute AggregateFinalizeExec"
holds_split_merge(&native_plan),
"the native plan must fold partial state in a Final node: {}",
datafusion::physical_plan::displayable(native_plan.as_ref()).indent(true)
);
let (_, native_rows) = native_labelled_rows(without_native).await;
let legacy = instance
@@ -3059,23 +3054,19 @@ async fn native_promql_sort_absent_and_histogram_quantile_match_legacy() {
let legacy_rows = legacy.iter().map(|batch| batch.num_rows()).sum::<usize>();
assert_eq!(native_rows, legacy_rows);
assert!(native_rows >= 1, "histogram_quantile must produce data");
assert_eq!(native_columns.len(), legacy_columns.len());
assert_eq!(native_value_index, legacy_value_index);
for (index, (native_column, legacy_column)) in
native_columns.iter().zip(legacy_columns).enumerate()
{
if index == native_value_index {
assert_ne!(native_column.name, legacy_column.name);
assert_eq!(native_column.name, "val");
assert_eq!(
legacy_column.name,
"prom_rate(ts_range,val,ts,Int64(20000))"
);
} else {
assert_eq!(native_column.name, legacy_column.name);
assert_eq!(native_column.data_type, legacy_column.data_type);
}
}
assert_eq!(
vec!["ts", "val", "host"],
native_columns
.iter()
.map(|column| column.name.as_str())
.collect::<Vec<_>>(),
"{shape}: native boundary columns"
);
assert_eq!(native_value_index, 1, "{shape}: native value position");
assert_eq!(
legacy_columns[legacy_value_index].name,
"prom_rate(ts_range,val,ts,Int64(20000))"
);
assert_eq!(
native_columns[native_value_index].data_type,
legacy_columns[legacy_value_index].data_type
@@ -3084,11 +3075,16 @@ async fn native_promql_sort_absent_and_histogram_quantile_match_legacy() {
.iter()
.flat_map(|batch| {
let timestamps = batch
.column(0)
.column_by_name("ts")
.unwrap()
.as_any()
.downcast_ref::<TimestampMillisecondArray>()
.unwrap();
let labels = cast(batch.column(1).as_ref(), &DataType::Utf8).unwrap();
let labels = cast(
batch.column_by_name("host").unwrap().as_ref(),
&DataType::Utf8,
)
.unwrap();
let labels = labels.as_any().downcast_ref::<StringArray>().unwrap();
let values = cast(
batch.column(native_value_index).as_ref(),
@@ -3111,11 +3107,16 @@ async fn native_promql_sort_absent_and_histogram_quantile_match_legacy() {
.iter()
.flat_map(|batch| {
let timestamps = batch
.column(0)
.column_by_name("ts")
.unwrap()
.as_any()
.downcast_ref::<TimestampMillisecondArray>()
.unwrap();
let labels = cast(batch.column(1).as_ref(), &DataType::Utf8).unwrap();
let labels = cast(
batch.column_by_name("host").unwrap().as_ref(),
&DataType::Utf8,
)
.unwrap();
let labels = labels.as_any().downcast_ref::<StringArray>().unwrap();
let values = cast(
batch.column(legacy_value_index).as_ref(),
@@ -5790,15 +5791,21 @@ async fn native_promql_metric_same_logical_table_offset_pair_matches_legacy() {
assert_eq!(native_rows, legacy_rows);
}
/// What every name of the frontend-side merge a split aggregate runs in front of contains.
const FINALIZE_EXEC_NAME: &str = "AggregateFinalize";
/// Whether any node of `plan` is the merge a split aggregate runs in front of.
/// Whether a Final fused node routes its aggregate state through the series repartitioner.
fn holds_split_merge(plan: &Arc<dyn datafusion::physical_plan::ExecutionPlan>) -> bool {
let mut stack = vec![plan.clone()];
while let Some(node) = stack.pop() {
if node.name().contains(FINALIZE_EXEC_NAME) {
return true;
let display = datafusion::physical_plan::displayable(node.as_ref())
.indent(false)
.to_string();
if node.name() == "FusedSeriesExec" && display.contains(" final") {
let mut descendants = node.children().into_iter().cloned().collect::<Vec<_>>();
while let Some(child) = descendants.pop() {
if child.name() == "SeriesRepartitionExec" {
return true;
}
descendants.extend(child.children().into_iter().cloned());
}
}
stack.extend(node.children().into_iter().cloned());
}
@@ -5855,14 +5862,16 @@ async fn native_promql_metric_two_aggregate_pair_matches_legacy() {
value,
)
};
for (query, expected) in [
for (query, expected, expected_columns) in [
(
"sum(native_pair_demo) / sum(native_pair_other)",
vec![(0, std::collections::BTreeMap::new(), 0.375)],
vec!["ts", "val"],
),
(
"sum by (host) (native_pair_demo) / sum by (host) (native_pair_other)",
vec![grouped("a", 0.25), grouped("b", 0.5)],
vec!["ts", "val", "host"],
),
] {
let sql = format!("TQL EVAL (0, 0, '1s') {query}");
@@ -5886,22 +5895,56 @@ async fn native_promql_metric_two_aggregate_pair_matches_legacy() {
// per region. Each segment binds its own source's metric, which is what makes the division
// read two different metrics rather than one read twice.
observer.clear();
let (native_columns, native_rows) =
native_labelled_rows(native_tql_output(&instance, &sql, true).await).await;
let root = observer.root_plan();
let native_output = native_tql_output(&instance, &sql, true).await;
// Check the actual output schema before the row reader sorts the columns.
let schema = match &native_output.data {
OutputData::Stream(stream) => stream.schema(),
OutputData::RecordBatches(batches) => batches.schema(),
_ => unreachable!("TQL EVAL returns record batches"),
};
let boundary_columns = schema
.column_schemas()
.iter()
.map(|column| column.name.as_str())
.collect::<Vec<_>>();
assert_eq!(
root.name(),
"FusedSeriesExec",
"{query}: the parent is the frontend's own: {}",
datafusion::physical_plan::displayable(root.as_ref()).indent(true)
expected_columns, boundary_columns,
"{query}: native boundary columns"
);
let (native_columns, native_rows) = native_labelled_rows(native_output).await;
let root = observer.root_plan();
let fused_parent = if root.name() == "ProjectionExec" {
let children = root.children();
assert_eq!(
1,
children.len(),
"{query}: the naming projection renames one input: {}",
datafusion::physical_plan::displayable(root.as_ref()).indent(true)
);
assert_eq!(
"FusedSeriesExec",
children[0].name(),
"{query}: the parent under the naming projection is the frontend's own: {}",
datafusion::physical_plan::displayable(root.as_ref()).indent(true)
);
Arc::clone(children[0])
} else {
assert_eq!(
root.name(),
"FusedSeriesExec",
"{query}: the parent is the frontend's own: {}",
datafusion::physical_plan::displayable(root.as_ref()).indent(true)
);
Arc::clone(&root)
};
assert_eq!(root.output_partitioning().partition_count(), 1, "{query}");
assert_eq!(
root.children().len(),
fused_parent.children().len(),
2,
"{query}: one input per segmented prefix: {}",
datafusion::physical_plan::displayable(root.as_ref()).indent(true)
);
let root = fused_parent;
// Each region answers one prefix: a repartition over the segment that evaluates the
// selector, over the scan bound to that selector's metric.
fn scan_metric(plan: &Arc<dyn datafusion::physical_plan::ExecutionPlan>) -> Option<String> {
@@ -5954,14 +5997,6 @@ async fn native_promql_metric_two_aggregate_pair_matches_legacy() {
legacy_columns.iter().any(|column| column == "ts"),
"{query}"
);
assert!(
native_columns.iter().any(|column| column == "ts"),
"{query}"
);
assert!(
native_columns.iter().any(|column| column == "val"),
"{query}: {native_columns:?}"
);
let labels = native_rows
.first()
.map(|(_, labels, _)| labels.keys().cloned().collect::<Vec<_>>())
@@ -8146,11 +8181,7 @@ async fn native_promql_cross_table_aggregate_offsets_keep_independent_scan_bound
let mut stack = vec![legacy_root.clone()];
while let Some(node) = stack.pop() {
assert_ne!(node.name(), "FusedSeriesExec", "legacy plan must not fuse");
assert_ne!(
node.name(),
"AggregateFinalizeExec",
"legacy plan must not finalize"
);
stack.extend(node.children().into_iter().cloned());
}
assert_eq!(legacy_rows, vec![(60_000, 20.0), (120_000, 15.0)]);
@@ -8294,11 +8325,7 @@ async fn native_promql_cross_table_range_aggregate_segments_keep_windows() {
let mut stack = vec![legacy_root];
while let Some(node) = stack.pop() {
assert_ne!(node.name(), "FusedSeriesExec", "legacy plan must not fuse");
assert_ne!(
node.name(),
"AggregateFinalizeExec",
"legacy plan must not finalize"
);
stack.extend(node.children().into_iter().cloned());
}
assert_eq!(legacy_rows, vec![(60_000, 2.0), (120_000, 4.0 / 3.0)]);
@@ -8932,12 +8959,12 @@ async fn native_promql_corpus_route_matrix() {
})
.collect::<Vec<_>>();
let fused = !fused_nodes_in_root(&root).is_empty();
let split = root_display.contains("AggregateFinalizeExec")
let split = holds_split_merge(&root)
&& regions.iter().any(|p| {
let d = datafusion::physical_plan::displayable(p.as_ref())
.indent(true)
.to_string();
d.contains("FusedSeriesExec") && d.contains("partial state:")
d.contains("FusedSeriesExec") && d.contains(" partial")
});
// `segmented` means the frontend's native node consumes what a region's native node
// produced. A `MergeScan` alone does not show that: its region side can be a plain scan
@@ -9513,7 +9540,7 @@ async fn native_promql_root_canonical_merge_keeps_the_scan_width() {
"the fold is a complete aggregate over the assembled series: {pair_sum_display}"
);
assert!(
!pair_sum_display.contains("AggregateFinalizeExec"),
!holds_split_merge(&pair_sum_plan),
"the fold is not planned as partition-local partial state: {pair_sum_display}"
);
let (_, pair_sum_rows) = native_labelled_rows(pair_sum).await;
@@ -9588,7 +9615,7 @@ async fn native_promql_root_canonical_merge_keeps_the_scan_width() {
"{shape}: the fold is a complete aggregate over the assembled frames: {display}"
);
assert!(
!display.contains("AggregateFinalizeExec"),
!holds_split_merge(&plan),
"{shape}: the aggregate runs over the boundary itself, not as partial state: {display}"
);
}
@@ -9683,7 +9710,7 @@ async fn native_promql_root_canonical_merge_keeps_the_scan_width() {
"the fold runs natively as a complete aggregate over the boundary: {sum_display}"
);
assert!(
!sum_display.contains("AggregateFinalizeExec"),
!holds_split_merge(&sum_plan),
"the fold's state is not the substitute for the canonical merge: {sum_display}"
);
assert!(