mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-17 16:24:07 +00:00
feat(promql): run subqueries through the native prefix (#10085)
* feat(promql): run subqueries through the native prefix A subquery evaluates its inner expression on its own grid, so the native route now plans it instead of sending every statement containing one to the existing planner. - `collect_selectors` recurses into a subquery's expression, so its selectors are prepared like any other and each gets its scan bounds from `raw_scan_bounds`. - `calls_rate_or_increase` recurses too, so a delta-temporality table still keeps a rate or an increase inside a subquery on the existing planner. - A window whose input is already evaluated - a subquery - is admitted above the split aggregate when its declared grid equals its direct input's grid, and the suffix is rebuilt with it. A raw-sample window still cannot sit above the cut. - A `FusedSeries` whose root is such a window publishes on that window's output grid, which is not necessarily the statement's; the binding follows it at every construction site, and the split's partial state keeps the cut aggregate's own grid. An evaluated-input window does not need an owner router for its sources, and a segment whose root is one is cut on its own grid. Verified with `-p promql`, `-p query`, `tests-integration --lib` (native_promql) and clippy. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep a subquery's aggregate operand and gate an evaluated cut Two shapes reached the native route and failed while planning instead of keeping the plan they had. A hybrid-binary operand folds a group one subquery down (`avg_over_time((sum by (job, host) (pm))[5m:1m]) + on(job, host) pn`): `single_selector_operand` counts the subquery's selector, so the operand looked admissible, while `contains_aggregate_expression` stopped at `PromExpr::Subquery` and never saw the aggregate. The admission then built a `FusedSeries` the physical planner could not convert (`No installed planner was able to convert the custom node to an execution plan ...`). The walk descends into a subquery again, and both statements keep the existing plan with the same rows. A window over an evaluated input was cut as a segment regardless of locality. For any such window whose input is not a bare `Input` - a raw window, a `label_replace` - `reads_across_series` is true and `pointwise_operand` false, so the `FusedSeries` arm of the dist analyzer answers `NonCommutative`: the segment stays above the `MergeScan` and nothing converts it (`sum by (job) (max_over_time(rate(pm[2m])[5m:1m]))` and two neighbours failed that way at instant and range evaluation). `region_local_segment` mirrors that arm's decision from the tree, and `cut_as_segment` declines an evaluated-input window it refuses, cutting the per-series prefix inside the window instead; where there is none, the statement keeps the plan it had. The analyzer is not extended: every range function but `last_over_time`/`first_over_time` drops the metric name, so such a window is genuinely cross-series and a region-local segment would publish two series under one label set. `build_evaluated_segment` stays, and so does its caller. The canonical pass inserts a merge under a label rewrite, and that boundary is only ever built from the frames the loop has published, so the cut needs the fallback; deleting it took every rewritten fold off the route, which is what `sum(label_replace(...))` and the histogram-over-rewrite boundary caught. The frames-bound segment itself is not the defect - the gate is: the evaluated-input window is declined before any cut is taken, so the fallback only sees cuts the analyzer places. Tests that asserted the older rejection are decided on evidence: `sum_over_time(rate(m[5m])[1h:])` is admitted now and its rows match the existing plan, so the plan-level test asserts the route and the integration case proves the rows; the aggregate-subquery case in `native_aggregate_complete_over_four_nonempty_mito_partitions` is answered by the split (one `Partial` producer, one frontend merge) and returns the same 15.0 on both routes, so it moved to the split cases rather than being dropped. Two fold-over-subquery shapes in `native_promql_subqueries_match_legacy` (`sum by (job) (max_over_time(rate(...)[5m:1m]))` and the fold above it) are answered natively for the same reason - the fallback builds their cut - and their rows equal the existing plan's, so they assert the route instead of the fallback; the rewrite shape still declines, because its window reads across series and there is no per-series prefix inside it to take instead. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert anchored subqueries do not re-fold across steps Signed-off-by: discord9 <discord9@163.com> * test(promql): drop the unused fallback-case helper Signed-off-by: discord9 <discord9@163.com> * test(promql): update subquery expectations for the native route Signed-off-by: discord9 <discord9@163.com> * fix(promql): publish subquery segments on their declared grid Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop unreachable segment arms and pin subquery routes Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
@@ -1779,7 +1779,13 @@ fn tree_inputs(tree: &NativeTree) -> Vec<&crate::native::plan::BoundInput> {
|
||||
fn partitioned_window_sources(node: &NativeNode) -> DataFusionResult<Vec<usize>> {
|
||||
fn visit(node: &NativeNode, sources: &mut Vec<usize>) -> DataFusionResult<()> {
|
||||
match &node.expr {
|
||||
NativeExpr::Window { input, .. } => {
|
||||
NativeExpr::Window { input, op } => {
|
||||
// A subquery window reads an evaluated input, whose fragments belong to one series
|
||||
// wherever they were produced, so it needs no owner of its own. A raw window
|
||||
// anywhere below it still does, so the walk continues into the input.
|
||||
if op.input_grid.is_some() {
|
||||
return visit(input, sources);
|
||||
}
|
||||
let NativeExpr::Input(bound) = &input.expr else {
|
||||
return Err(DataFusionError::Plan(
|
||||
"partitioned FusedSeries windows require a direct Input under Window"
|
||||
|
||||
@@ -70,8 +70,10 @@ pub(crate) fn histogram_group_is_single(group: &Grouping) -> bool {
|
||||
/// the merged values. `topk`, a second aggregate and vector-vector matching still need a whole-vector
|
||||
/// boundary.
|
||||
///
|
||||
/// An instant selection or window above the cut cannot be applied to the merged values under the
|
||||
/// tree's current binding: the cut output is evaluated, whereas these nodes consume raw samples.
|
||||
/// An instant selection or a window over raw samples above the cut cannot be applied to the merged
|
||||
/// values under the tree's current binding: the cut output is evaluated, whereas these nodes consume
|
||||
/// raw samples. A subquery window instead reads an evaluated input, so it is admitted when its
|
||||
/// declared grid equals its direct input's grid.
|
||||
pub(crate) fn split(
|
||||
node: &NativeNode,
|
||||
fields: &[FieldSpec],
|
||||
@@ -228,6 +230,34 @@ pub(crate) fn split(
|
||||
..
|
||||
} => Err(declined("a whole-vector operation above a split aggregate")),
|
||||
NativeExpr::Merge { .. } => Err(declined("an aggregate split across a canonical merge")),
|
||||
// A window whose input is already evaluated - a subquery - applies to the merged values the
|
||||
// suffix reads, provided it evaluates on that input's own grid. The declared grid must equal
|
||||
// its direct input's grid; the cut's grid need not be that grid, because a window between
|
||||
// them may already have re-gridded the values. A raw-sample window cannot sit above the cut.
|
||||
NativeExpr::Window { input, op } => {
|
||||
let Some(mut inner) = split(input, fields, time_index)? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let input_grid = input
|
||||
.output
|
||||
.samples()
|
||||
.and_then(|schema| schema.eval_grid.as_ref());
|
||||
if op.input_grid.as_ref().is_none_or(|declared| {
|
||||
input_grid.is_none_or(|input| declared.spec() != input.spec())
|
||||
}) {
|
||||
return Err(declined(
|
||||
"a window above an aggregate with a different evaluation grid",
|
||||
));
|
||||
}
|
||||
inner.suffix = NativeNode {
|
||||
expr: NativeExpr::Window {
|
||||
input: Box::new(inner.suffix),
|
||||
op: op.clone(),
|
||||
},
|
||||
output: node.output.clone(),
|
||||
};
|
||||
Ok(Some(inner))
|
||||
}
|
||||
// These read values the merged state does not answer: an instant selection or window reads
|
||||
// raw samples, `scalar`/`vector` answers one value or one series instead of the vector, and a
|
||||
// parameterized map reads the whole vector at a point. `ParameterizedTopK` declines here
|
||||
@@ -235,7 +265,6 @@ pub(crate) fn split(
|
||||
// a bound one does, so it needs the same whole-vector boundary. Admitting the one and
|
||||
// refusing the other would be an inconsistency rather than a capability.
|
||||
NativeExpr::Instant { input, .. }
|
||||
| NativeExpr::Window { input, .. }
|
||||
| NativeExpr::Timestamp { input }
|
||||
| NativeExpr::Vector { input, .. }
|
||||
| NativeExpr::VectorOf { input }
|
||||
@@ -541,9 +570,9 @@ impl Split {
|
||||
|
||||
/// Puts `merged` at the cut and returns the suffix above it.
|
||||
///
|
||||
/// A suffix is the spine of point-wise maps, constant-sided binaries and at most one histogram
|
||||
/// operation between the cut and the root, with scalar constants and the evaluation time preserved.
|
||||
/// Other whole-vector operations are refused by `split`.
|
||||
/// A suffix is the spine of point-wise maps, constant-sided binaries, evaluated-input windows and at
|
||||
/// most one histogram operation between the cut and the root, with scalar constants and the
|
||||
/// evaluation time preserved. Other whole-vector operations are refused by `split`.
|
||||
fn enter_suffix(node: NativeNode, merged: &NativeNode) -> DfResult<NativeNode> {
|
||||
let NativeNode { expr, output } = node;
|
||||
let rejected = || {
|
||||
@@ -557,6 +586,11 @@ fn enter_suffix(node: NativeNode, merged: &NativeNode) -> DfResult<NativeNode> {
|
||||
NativeExpr::Scalar(_) | NativeExpr::EvalTime { .. } => {
|
||||
return Ok(NativeNode { expr, output });
|
||||
}
|
||||
// A subquery window above the cut is rebuilt over the merged values, on its own grid.
|
||||
NativeExpr::Window { input, op } => NativeExpr::Window {
|
||||
input: Box::new(enter_suffix(*input, merged)?),
|
||||
op,
|
||||
},
|
||||
NativeExpr::Map { input, op } => NativeExpr::Map {
|
||||
input: Box::new(enter_suffix(*input, merged)?),
|
||||
op,
|
||||
|
||||
@@ -1279,12 +1279,13 @@ impl PromPlanner {
|
||||
let Some(rawinput) = children.into_iter().next() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let publish_grid = fused_publish_grid(final_tree.root(), &grid);
|
||||
let node = plan_split_aggregate(
|
||||
Arc::from(prom_expr.to_string()),
|
||||
partial,
|
||||
final_tree,
|
||||
FusedSeriesBinding {
|
||||
grid,
|
||||
grid: publish_grid,
|
||||
time_index: time.clone(),
|
||||
tags,
|
||||
metric_name: metric,
|
||||
@@ -1374,11 +1375,12 @@ impl PromPlanner {
|
||||
} else {
|
||||
(tags, metric)
|
||||
};
|
||||
let publish_grid = fused_publish_grid(tree.root(), &grid);
|
||||
let node = FusedSeries::try_new(
|
||||
Arc::from(prom_expr.to_string()),
|
||||
tree,
|
||||
FusedSeriesBinding {
|
||||
grid,
|
||||
grid: publish_grid,
|
||||
time_index: time,
|
||||
tags,
|
||||
metric_name: metric,
|
||||
@@ -1631,15 +1633,49 @@ fn exportable(node: &NativeNode) -> bool {
|
||||
.is_some_and(|schema| schema.eval_grid.is_some())
|
||||
}
|
||||
|
||||
/// Whether `node` is a window over an evaluated input - the shape a subquery over anything but a
|
||||
/// bare selection compiles to.
|
||||
fn evaluated_input_window(node: &NativeNode) -> bool {
|
||||
matches!(&node.expr, NativeExpr::Window { op, .. } if op.input_grid.is_some())
|
||||
}
|
||||
|
||||
/// Whether the dist analyzer places a segment over `node` region-locally.
|
||||
///
|
||||
/// A segment is a `FusedSeries` node the analyzer has to move below the `MergeScan` for the physical
|
||||
/// planner to read the scans it names. The `FusedSeries` arm of `Categorizer::check_extension_plan`
|
||||
/// (`src/query/src/dist_plan/commutativity.rs`) is what decides that, from the bound tree alone: a
|
||||
/// root that [`reads_across_series`] is `NonCommutative` unless it is a per-series evaluation or the
|
||||
/// `Partial` half of a split fold. A segment it refuses stays above the merge, where the physical
|
||||
/// planner finds neither a region scan nor an explicit-label frame below it, so the statement fails
|
||||
/// while being planned instead of keeping the plan it had. Segmentation therefore asks that question
|
||||
/// before it cuts, and [`cut_as_segment`] declines the nodes this refuses.
|
||||
fn region_local_segment(node: &NativeNode) -> bool {
|
||||
!reads_across_series(node) || pointwise_operand(node) || independent_scan_pair(node)
|
||||
}
|
||||
|
||||
/// Whether `node` becomes a segment of its own under a parent that must gather.
|
||||
///
|
||||
/// Cut local prefixes, global consumers and canonical merges separately so the prefix keeps its
|
||||
/// scan width, and every global consumer sees the canonical boundary below it. A pointwise wrapper
|
||||
/// of a global consumer is cut as well, so it cannot hide that consumer from recursive lowering.
|
||||
fn cut_as_segment(node: &NativeNode, canonical_segmentation: bool) -> bool {
|
||||
// A window over an evaluated input answers per series on its own grid, so its own shape says
|
||||
// nothing about where a segment over it can go: it drops the metric name, so two of the series
|
||||
// it answers can end under one label set, and the dist analyzer reads it as cross-series
|
||||
// ([`region_local_segment`]). Cutting one where the analyzer will not place it leaves a
|
||||
// `FusedSeries` above the `MergeScan` that the physical planner cannot build, so the segment is
|
||||
// only taken where the analyzer places it region-locally; otherwise the per-series prefix
|
||||
// inside the window is cut instead, and where there is none the statement keeps the plan it had.
|
||||
if evaluated_input_window(node) && !region_local_segment(node) {
|
||||
return false;
|
||||
}
|
||||
// A window over an evaluated input is answered per series on its own grid, so it is not the
|
||||
// global gather a raw window's locality reports.
|
||||
let evaluated_window = evaluated_input_window(node);
|
||||
if !exportable(node)
|
||||
|| matches!(node.expr, NativeExpr::Input(_))
|
||||
|| (!canonical_segmentation
|
||||
&& !evaluated_window
|
||||
&& matches!(input_locality(node), InputLocality::SinglePartition))
|
||||
{
|
||||
return false;
|
||||
@@ -1873,6 +1909,21 @@ fn cut_segments(node: &NativeNode, canonical_segmentation: bool) -> Segment {
|
||||
}
|
||||
}
|
||||
|
||||
/// The grid one `FusedSeries` publishes on: the grid its root declares it produces, or the
|
||||
/// statement's grid when the root declares none.
|
||||
///
|
||||
/// A root that produces one value per grid point says which grid that is, and that declared grid is
|
||||
/// the one its consumer reads it on: a window over an evaluated input - a subquery - produces on the
|
||||
/// window's own output grid, and a non-window root inside such a window, like the `Map` an
|
||||
/// expression of a subquery compiles to, produces on the subquery's grid rather than on the
|
||||
/// statement's. A statement root declares the statement's grid, so the two agree there. A root with
|
||||
/// no declared grid - raw samples - is read where it was scanned and keeps the caller's grid.
|
||||
fn fused_publish_grid(root: &NativeNode, statement_grid: &Arc<EvalGrid>) -> Arc<EvalGrid> {
|
||||
root.output
|
||||
.grid()
|
||||
.map_or_else(|| Arc::clone(statement_grid), Arc::clone)
|
||||
}
|
||||
|
||||
/// Lowers a tree whose root is a canonical merge into a producer and a canonical router.
|
||||
///
|
||||
/// The tree says the statement answers one series per canonical label set; what it cannot say is
|
||||
@@ -2120,11 +2171,12 @@ fn build_evaluated_segment(
|
||||
}
|
||||
let frame = FusedSeries::series_frame_schema(&tree).ok()?;
|
||||
let frame = DFSchema::try_from(frame.as_ref().clone()).ok()?;
|
||||
let publish_grid = fused_publish_grid(tree.root(), grid);
|
||||
let plan = FusedSeries::try_new_with_format(
|
||||
Arc::from("segment"),
|
||||
tree,
|
||||
FusedSeriesBinding {
|
||||
grid: Arc::clone(grid),
|
||||
grid: publish_grid,
|
||||
time_index: time_index.to_string(),
|
||||
tags: output_tags.clone(),
|
||||
metric_name: None,
|
||||
@@ -2168,6 +2220,11 @@ fn build_inner_segment(
|
||||
.collect::<Option<Vec<_>>>()?;
|
||||
let tree = NativeTree::try_new(rewrite_inner_segment(node, &sources)).ok()?;
|
||||
let output_tags = native_output_tags(tree.root(), &source_tags)?;
|
||||
// A segment publishes on the grid its own root declares, which is the grid its parent reads it
|
||||
// on: the segments of a subquery's inner expression are evaluated on the subquery's grid, not on
|
||||
// the statement's, whatever their root computes. The statement's grid is only the fallback for a
|
||||
// root that declares none.
|
||||
let segment_grid = fused_publish_grid(tree.root(), grid);
|
||||
let (inputs, metric, time_index, tags) = if let [index] = sources.as_slice() {
|
||||
let (metric, time_index, tags) = per_source.get(*index)?;
|
||||
(scans, metric.clone(), time_index.clone(), tags.clone())
|
||||
@@ -2209,7 +2266,7 @@ fn build_inner_segment(
|
||||
Arc::from("segment"),
|
||||
tree,
|
||||
FusedSeriesBinding {
|
||||
grid: Arc::clone(grid),
|
||||
grid: segment_grid,
|
||||
time_index: time_index.clone(),
|
||||
tags: tags.clone(),
|
||||
metric_name: metric.clone(),
|
||||
@@ -2251,7 +2308,15 @@ fn segment_tree(
|
||||
root_kind: FieldKind,
|
||||
remote_raw_boundary: bool,
|
||||
) -> Result<Option<LogicalPlan>> {
|
||||
if !matches!(input_locality(tree.root()), InputLocality::SinglePartition) {
|
||||
// Only a tree the analyzer gathers is worth splitting: its per-series prefixes then run where
|
||||
// their input is read. A window over an evaluated input can need that split even where
|
||||
// [`input_locality`] reports a local tree - it drops the metric name, so [`reads_across_series`]
|
||||
// reads it as cross-series - and the per-series prefix inside it is what keeps the partitions it
|
||||
// scanned. Proceeding only admits the attempt: every cut is admitted by [`cut_as_segment`]
|
||||
// individually, which is where the analyzer's placement is asked.
|
||||
if !matches!(input_locality(tree.root()), InputLocality::SinglePartition)
|
||||
&& !evaluated_input_window(tree.root())
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
// A segment boundary has to carry something, and the only thing it can carry is a per-series
|
||||
@@ -2259,6 +2324,10 @@ fn segment_tree(
|
||||
// such subtree inside it has nothing to cut, and cutting anything else - an aggregate's own
|
||||
// state, a constant - is either impossible or a plain recomputation of the same work in one
|
||||
// more node. Cutting on locality alone would accept one here.
|
||||
//
|
||||
// The window this tree may be rooted at is not one of those consumers, but it is no boundary
|
||||
// either: `cut_as_segment` never cuts one, and what keeps its input's partitions is the
|
||||
// per-series prefix below it, which the cuts below are admitted for.
|
||||
// A grouping by `__name__` keeps that label while the fused node's declared columns cannot
|
||||
// name it. With multiple sources, groups differing only by metric name would therefore be
|
||||
// reported under one label set. A single-source tree cannot merge distinct metric-name groups
|
||||
@@ -2273,10 +2342,13 @@ fn segment_tree(
|
||||
let mut first_iteration = true;
|
||||
let mut changed = false;
|
||||
loop {
|
||||
// A tree the analyzer keeps partition-local needs no segment; a window over an evaluated
|
||||
// input is still worth attempting one for, so its per-series prefix keeps its partitions.
|
||||
if !matches!(
|
||||
input_locality(current_tree.root()),
|
||||
InputLocality::SinglePartition
|
||||
) {
|
||||
) && !evaluated_input_window(current_tree.root())
|
||||
{
|
||||
break;
|
||||
}
|
||||
let segment = cut_segments(current_tree.root(), canonical_segmentation);
|
||||
@@ -2339,6 +2411,12 @@ fn segment_tree(
|
||||
if built.is_none() && contains_global_consumer(cut) {
|
||||
return Ok(None);
|
||||
}
|
||||
// A cut the first iteration cannot build from the scans is built from the frames this
|
||||
// loop has already published: the canonical pass inserts a merge under a label rewrite,
|
||||
// and that boundary - rebuilt here as a producer, a frame and a router - is what answers
|
||||
// one series per label set across partitions. Which cuts are taken is
|
||||
// [`cut_as_segment`]'s decision, and it declines the evaluated-input window the analyzer
|
||||
// will not place region-locally, so this fallback only ever sees the cuts it admitted.
|
||||
let built = built.or_else(|| {
|
||||
build_evaluated_segment(
|
||||
prom_expr,
|
||||
@@ -2472,11 +2550,12 @@ fn segment_tree(
|
||||
)
|
||||
};
|
||||
let (segment_metric, segment_time_index) = segment_binding(¤t_source_ids);
|
||||
let publish_grid = fused_publish_grid(parent.root(), grid);
|
||||
let Ok(plan) = FusedSeries::try_new_with_format(
|
||||
Arc::from(prom_expr.to_string()),
|
||||
parent,
|
||||
FusedSeriesBinding {
|
||||
grid: Arc::clone(grid),
|
||||
grid: publish_grid,
|
||||
time_index: segment_time_index,
|
||||
tags: tags.to_vec(),
|
||||
metric_name: segment_metric,
|
||||
@@ -2604,10 +2683,20 @@ fn plan_split_aggregate(
|
||||
)))
|
||||
.context(DataFusionPlanningSnafu);
|
||||
}
|
||||
// The partial tree publishes aggregate state on the cut aggregate's grid, which is not
|
||||
// necessarily the statement's when the suffix re-grids the values - a subquery above the cut
|
||||
// produces on its own grid. The final tree's binding is the one the caller derived for it.
|
||||
let partial_binding = FusedSeriesBinding {
|
||||
grid: match &partial.root().output {
|
||||
NativeType::AggregateState(state) => Arc::clone(&state.grid),
|
||||
_ => Arc::clone(&binding.grid),
|
||||
},
|
||||
..binding.clone()
|
||||
};
|
||||
let partial = FusedSeries::try_new_state_export(
|
||||
Arc::clone(&promql),
|
||||
partial,
|
||||
binding.clone(),
|
||||
partial_binding,
|
||||
vec![rawinput],
|
||||
)
|
||||
.context(DataFusionPlanningSnafu)?;
|
||||
@@ -3236,6 +3325,12 @@ fn contains_aggregate_expression(expr: &PromExpr) -> bool {
|
||||
.args
|
||||
.iter()
|
||||
.any(|arg| contains_aggregate_expression(arg)),
|
||||
// A subquery evaluates its expression on its own grid, but a fold inside it still folds a
|
||||
// group no single region owns, so an operand holding one needs the split route just as a
|
||||
// direct aggregate does. [`single_selector_operand`] counts a subquery's selector as the
|
||||
// operand's one selector, so this is what keeps a subquery around an aggregate out of the
|
||||
// join.
|
||||
PromExpr::Subquery(subquery) => contains_aggregate_expression(&subquery.expr),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
@@ -3326,7 +3421,11 @@ fn collect_selectors<'a>(
|
||||
None => true,
|
||||
};
|
||||
}
|
||||
PromExpr::Subquery(_) => return false,
|
||||
// A subquery evaluates its expression on its own grid; its selectors are prepared like any
|
||||
// other, each with the scan bounds `raw_scan_bounds` derives from its own range and step.
|
||||
PromExpr::Subquery(subquery) => {
|
||||
return collect_selectors(stmt, &subquery.expr, out);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
true
|
||||
@@ -4139,6 +4238,12 @@ mod tests {
|
||||
// the shapes that produced a `tag_0` planning error before the aggregate gate existed.
|
||||
"sum(A) / B",
|
||||
"A / sum(B)",
|
||||
// The same fold one subquery down. The subquery's selector is the operand's one
|
||||
// selector, so only the aggregate gate keeps this operand out of the join - without it
|
||||
// the admitted operand fails at execution, where no planner can build the fused node
|
||||
// over the gathered subquery.
|
||||
"avg_over_time((sum by (job, host) (A))[5m:1m]) + on(job, host) B",
|
||||
"A + on(job, host) avg_over_time((sum by (job, host) (B))[5m:1m])",
|
||||
] {
|
||||
assert!(!hybrid_admission(query), "{query}");
|
||||
}
|
||||
@@ -4154,6 +4259,30 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// An aggregate inside a subquery is still an aggregate, so a hybrid operand that holds one is
|
||||
/// not the per-series evaluation the join needs.
|
||||
///
|
||||
/// This is the admission decision behind a statement that used to fail while executing: both
|
||||
/// shapes below were admitted, and the operand's subquery then had to be gathered for a join
|
||||
/// that no installed planner can build. Refusing them keeps the plan the existing route builds.
|
||||
#[test]
|
||||
fn subquery_aggregate_operand_is_refused() {
|
||||
for query in [
|
||||
"avg_over_time((sum by (job, host) (A))[5m:1m]) + on(job, host) B",
|
||||
"A + on(job, host) avg_over_time((sum by (job, host) (B))[5m:1m])",
|
||||
] {
|
||||
assert!(!hybrid_admission(query), "{query}");
|
||||
}
|
||||
// The subquery on its own is admitted: what the gate refuses is the fold inside it, not the
|
||||
// subquery that carries it.
|
||||
for query in [
|
||||
"avg_over_time((A)[5m:1m]) + on(job, host) B",
|
||||
"A + on(job, host) avg_over_time((B)[5m:1m])",
|
||||
] {
|
||||
assert!(hybrid_admission(query), "{query}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn millisecond_scan_bounds_preserve_open_closed_and_maximum_edges() {
|
||||
let inclusive = scan_time_filter(0, 10, true, "ts").to_string();
|
||||
|
||||
@@ -3488,25 +3488,22 @@ async fn native_fused_depth_limit_falls_back_to_legacy_plan() {
|
||||
);
|
||||
}
|
||||
|
||||
/// What the compiler has no node for still keeps the existing plan.
|
||||
/// A window over an evaluated input enters the native route, and only when it is asked for.
|
||||
///
|
||||
/// This test used to assert the opposite: a subquery evaluates its own body on a grid of its own,
|
||||
/// which the route did not offer, so the statement kept the existing plan. The route compiles a
|
||||
/// subquery now, and the window's per-series prefix - the `rate` over the raw scan - is the segment
|
||||
/// it is cut as, so the statement is planned natively. What the answer is worth is checked row for
|
||||
/// row against the existing plan in `native_promql_subqueries_match_legacy` (tests-integration),
|
||||
/// which runs both routes over the same fixture.
|
||||
#[tokio::test]
|
||||
async fn native_fused_unsupported_shape_falls_back() {
|
||||
for query in [
|
||||
// A subquery evaluates its own body on a grid of its own, which this route does not offer.
|
||||
"sum_over_time(rate(some_metric[5m])[1h:])",
|
||||
] {
|
||||
async fn native_fused_subquery_window_enters_the_native_route() {
|
||||
for query in ["sum_over_time(rate(some_metric[5m])[1h:])"] {
|
||||
let plan = build_native_fused_plan(query, true).await;
|
||||
let display = plan.display_indent().to_string();
|
||||
assert!(
|
||||
!display.contains("FusedSeries"),
|
||||
"`{query}` must not enter the native route: {display}"
|
||||
);
|
||||
// Falling back is the existing plan, not an approximation of it.
|
||||
let without_flag = build_native_fused_plan(query, false).await;
|
||||
assert_eq!(
|
||||
display,
|
||||
without_flag.display_indent().to_string(),
|
||||
"`{query}` must keep the existing plan"
|
||||
display.contains("FusedSeries"),
|
||||
"`{query}` must enter the native route: {display}"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -5501,17 +5498,17 @@ async fn native_fused_preparation_decline_preserves_legacy_plan() {
|
||||
);
|
||||
}
|
||||
|
||||
/// A subquery is planned natively: it is a window over an evaluated input, which the split and
|
||||
/// the segment machinery both understand, so it no longer keeps the existing plan.
|
||||
#[tokio::test]
|
||||
async fn native_fused_subquery_falls_back_to_legacy_plan() {
|
||||
async fn native_fused_subquery_plans_natively() {
|
||||
for query in [
|
||||
"rate(some_metric[5m:1m])",
|
||||
"rate(rate(some_metric[5m:1m])[10m:1m])",
|
||||
] {
|
||||
let plan = build_native_fused_plan(query, true).await;
|
||||
let display = plan.display_indent().to_string();
|
||||
assert!(!display.contains("FusedSeries"), "{query}: {display}");
|
||||
let legacy = build_native_fused_plan(query, false).await;
|
||||
assert_eq!(display, legacy.display_indent().to_string(), "{query}");
|
||||
assert!(display.contains("FusedSeries"), "{query}: {display}");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -47,12 +47,10 @@
|
||||
//! satisfy. No other counter query reads that tag: an aggregate drops the labels it does not group
|
||||
//! by, so their answers are the ones the untagged fixture gives.
|
||||
//!
|
||||
//! A query the native route declines is checked the other way round. [`check_fallback_case`] runs
|
||||
//! shapes the compiler has no node for - `scalar(v)` and a subquery - with the native request on and
|
||||
//! with it off, and requires that no native node ran in either plan and that both runs report rows
|
||||
//! derived from the fixture by hand. A hint that quietly
|
||||
//! took the route for one of them would show a fused producer or a frontend merge in the plans that
|
||||
//! executed, which is what is observed.
|
||||
//! Two shapes that used to be declined are checked on the plans that really ran. `scalar(v)` reads
|
||||
//! the whole vector in one fused node and needs no merge; a subquery over a fold splits into one
|
||||
//! producer and one frontend merge. A hint that quietly kept the existing route would show no native
|
||||
//! node, which is what the plan observer checks, and both routes must answer the hand-derived rows.
|
||||
//!
|
||||
//! A binary operation whose sibling is a literal scalar is carried by the split on either side of
|
||||
//! the fold: below the aggregate it is computed in the per-series prefix (`sum(8 * rate(...))`,
|
||||
@@ -739,9 +737,16 @@ async fn native_hybrid_binary_05_06_08() {
|
||||
);
|
||||
let (legacy, legacy_capture) = run_query(&instance, &subquery, false, &observer).await;
|
||||
let (native, native_capture) = run_query(&instance, &subquery, true, &observer).await;
|
||||
// A subquery now runs on the native route, so the existing plan must hold no native node while
|
||||
// the native run must exercise one. The rows below are what says the route answers what the
|
||||
// existing plan does.
|
||||
assert!(
|
||||
legacy_capture.fused.is_empty() && native_capture.fused.is_empty(),
|
||||
"hybrid must be disabled inside subquery: {subquery}"
|
||||
legacy_capture.fused.is_empty(),
|
||||
"the existing route must not plan native nodes: {subquery}"
|
||||
);
|
||||
assert!(
|
||||
!native_capture.fused.is_empty(),
|
||||
"the subquery must run on the native route: {subquery}"
|
||||
);
|
||||
assert_eq!(
|
||||
legacy.rows.len(),
|
||||
@@ -1606,15 +1611,14 @@ fn assert_finalize_reads_region_state(
|
||||
}
|
||||
}
|
||||
|
||||
/// Checks one query the native route declines: it must stay declined on both routes, and both must
|
||||
/// answer what the fixture derives by hand.
|
||||
/// Checks one query the native route answers by splitting the aggregate below a subquery.
|
||||
///
|
||||
/// The native request is on for one run and off for the other, so the hint being asked for and the
|
||||
/// hint being absent are both exercised: neither plan may hold a fused producer or a frontend merge,
|
||||
/// and the rows - labels, evaluation time, values and their multiplicity - are the same hand-derived
|
||||
/// ones for both. The observer is asked to have seen a plan at all, because a run that handed nothing
|
||||
/// over would otherwise satisfy "no native node" without observing anything.
|
||||
async fn check_fallback_case(
|
||||
/// The per-series prefix runs in a `Partial` producer on every partition and the window above it
|
||||
/// finishes the answer in the `Final` half over the merged values, so the native run must show
|
||||
/// exactly one producer and one frontend merge - the opposite of a fallback case - while the existing
|
||||
/// plan shows neither. Both routes must answer what the fixture derives by hand, which is what says
|
||||
/// the split's answer is the statement's answer and not an approximation of it.
|
||||
async fn check_split_case(
|
||||
instance: &Arc<Instance>,
|
||||
observer: &NativePlanObserver,
|
||||
case: &AggregateCase,
|
||||
@@ -1630,13 +1634,32 @@ async fn check_fallback_case(
|
||||
"`{query}`: the {route} run handed no plan to the observer, so nothing about the plan \
|
||||
that ran was observed"
|
||||
);
|
||||
assert!(
|
||||
capture.fused.is_empty() && capture.finalize.is_empty(),
|
||||
"`{query}`: the {route} run must keep the existing plan, but the plans that ran held {} \
|
||||
fused producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
if native {
|
||||
assert_eq!(
|
||||
1,
|
||||
capture.fused.len(),
|
||||
"`{query}`: the {route} run must split the aggregate into one producer, but the \
|
||||
plans that ran held {} producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
assert_eq!(
|
||||
1,
|
||||
capture.finalize.len(),
|
||||
"`{query}`: the {route} run must finish the fold in one frontend merge, but the \
|
||||
plans that ran held {} producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
} else {
|
||||
assert!(
|
||||
capture.fused.is_empty() && capture.finalize.is_empty(),
|
||||
"`{query}`: the {route} run must keep the existing plan, but the plans that ran \
|
||||
held {} fused producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
}
|
||||
assert_rows(&result.rows, &case.expected, &format!("{route} `{query}`"));
|
||||
}
|
||||
}
|
||||
@@ -2481,20 +2504,6 @@ async fn native_aggregate_complete_over_four_nonempty_mito_partitions() {
|
||||
)
|
||||
.await;
|
||||
|
||||
// A shape the native compiler declines, checked on the plans that really ran: it may hold
|
||||
// neither a fused producer nor a frontend merge, and both routes must answer what the fixture
|
||||
// derives by hand. A subquery evaluates its own body on a grid of its own: it reads across
|
||||
// series and cannot be answered by a producer per partition, so it keeps the existing plan. The
|
||||
// subquery's window spans the two inner points `EVAL_SECS - 1` and `EVAL_SECS`, at each of
|
||||
// which the same extrapolated rate answers 15 - both samples stay further from the window's
|
||||
// ends than the 66s threshold - so the maximum is 15 as well.
|
||||
let fallback_case = case(
|
||||
format!("max_over_time((sum(rate({COUNTER_TABLE}[5m])))[2s:1s])"),
|
||||
&[(&[], 15.0)],
|
||||
false,
|
||||
);
|
||||
check_fallback_case(&instance, &observer, &fallback_case).await;
|
||||
|
||||
// `scalar` reads the whole vector and the native route answers it, so it is no longer one of
|
||||
// the declined shapes. `scalar` reports the one value present at a point only when exactly one
|
||||
// series is present at it, so the five counter series make it NaN; a route that folded one
|
||||
@@ -2540,6 +2549,20 @@ async fn native_aggregate_complete_over_four_nonempty_mito_partitions() {
|
||||
&legacy.rows,
|
||||
"native `scalar` against the existing plan",
|
||||
);
|
||||
// A subquery over a fold is not declined: the compiler binds a window over an evaluated input to
|
||||
// that input's own grid, so the split carries it - the per-series prefix runs in a `Partial`
|
||||
// producer on every partition and the window above it finishes the answer in the `Final` half
|
||||
// over the merged values. The subquery's window spans the two inner points `EVAL_SECS - 1` and
|
||||
// `EVAL_SECS`, at each of which the same extrapolated rate answers 15 - both samples stay further
|
||||
// from the window's ends than the 66s threshold - so the maximum is 15 as well.
|
||||
let split_cases = [case(
|
||||
format!("max_over_time((sum(rate({COUNTER_TABLE}[5m])))[2s:1s])"),
|
||||
&[(&[], 15.0)],
|
||||
false,
|
||||
)];
|
||||
for case in &split_cases {
|
||||
check_split_case(&instance, &observer, case).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Histogram functions above the cut merge Partial state before evaluating the complete buckets.
|
||||
|
||||
@@ -2028,7 +2028,11 @@ async fn native_site_rows(output: Output) -> Vec<(i64, String, String, f64)> {
|
||||
rows
|
||||
}
|
||||
|
||||
async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
/// The rows of a TQL answer, labelled by `label`.
|
||||
///
|
||||
/// `None` labels each row with every label column it carries, by name, so an answer whose label set
|
||||
/// is not the `host` one can be compared with the existing route's the same way.
|
||||
async fn native_rows(output: Output, label: Option<&str>) -> Vec<(i64, String, f64)> {
|
||||
let batches = match output.data {
|
||||
OutputData::Stream(stream) => collect_batches(stream).await.unwrap(),
|
||||
OutputData::RecordBatches(batches) => batches,
|
||||
@@ -2043,30 +2047,60 @@ async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
.as_any()
|
||||
.downcast_ref::<TimestampMillisecondArray>()
|
||||
.unwrap();
|
||||
let hosts = cast(
|
||||
batch.column_by_name("host").unwrap().as_ref(),
|
||||
&DataType::Utf8,
|
||||
)
|
||||
.unwrap();
|
||||
let hosts = hosts.as_any().downcast_ref::<StringArray>().unwrap();
|
||||
let value_columns = batch
|
||||
.columns()
|
||||
.iter()
|
||||
.filter(|column| column.data_type() == &DataType::Float64)
|
||||
.enumerate()
|
||||
.filter(|(_, column)| column.data_type() == &DataType::Float64)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(value_columns.len(), 1, "expected one Float64 value column");
|
||||
let values = value_columns[0]
|
||||
let (value_index, value_column) = value_columns[0];
|
||||
let labels = match label {
|
||||
Some(label) => {
|
||||
let labels = cast(
|
||||
batch.column_by_name(label).unwrap().as_ref(),
|
||||
&DataType::Utf8,
|
||||
)
|
||||
.unwrap();
|
||||
let labels = labels.as_any().downcast_ref::<StringArray>().unwrap();
|
||||
assert_eq!(labels.null_count(), 0);
|
||||
(0..batch.num_rows())
|
||||
.map(|row| labels.value(row).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
}
|
||||
// Every label column, by name: the value column and the time index are the two that
|
||||
// are not one, and a route names its value differently from the other.
|
||||
None => (0..batch.num_rows())
|
||||
.map(|row| {
|
||||
batch
|
||||
.schema
|
||||
.column_schemas()
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(index, column)| *index != value_index && column.name != "ts")
|
||||
.filter_map(|(_, column)| {
|
||||
let values = batch.column_by_name(&column.name)?.as_ref();
|
||||
let values = cast(values, &DataType::Utf8).ok()?;
|
||||
let values = values.as_any().downcast_ref::<StringArray>()?;
|
||||
(!values.is_null(row))
|
||||
.then(|| format!("{}={}", column.name, values.value(row)))
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join(",")
|
||||
})
|
||||
.collect::<Vec<_>>(),
|
||||
};
|
||||
let values = value_column
|
||||
.as_any()
|
||||
.downcast_ref::<Float64Array>()
|
||||
.unwrap();
|
||||
assert_eq!(timestamps.null_count(), 0);
|
||||
assert_eq!(hosts.null_count(), 0);
|
||||
assert_eq!(values.null_count(), 0);
|
||||
(0..batch.num_rows())
|
||||
.map(|row| {
|
||||
(
|
||||
timestamps.value(row),
|
||||
hosts.value(row).to_string(),
|
||||
labels[row].clone(),
|
||||
values.value(row),
|
||||
)
|
||||
})
|
||||
@@ -2077,6 +2111,10 @@ async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
rows
|
||||
}
|
||||
|
||||
async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
native_rows(output, Some("host")).await
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_clamp_aggregates_execute_on_native_segments() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
@@ -7248,38 +7286,41 @@ async fn native_promql_repeated_selector_binary_uses_independent_scans() {
|
||||
native_result_rows(native_tql_output(&instance, &different_matchers, false).await).await;
|
||||
assert_eq!(different_matcher_rows, different_matcher_legacy);
|
||||
|
||||
// Group matching executes natively; the subquery case remains on the ordinary path, unchanged.
|
||||
let grouped_expected = expected[..HOSTS].to_vec();
|
||||
for (index, (sql, expected_rows)) in [
|
||||
(
|
||||
format!(
|
||||
r#"TQL EVAL (0, 0, '1s') {table}{{host!=""}} - on(host) group_left() ({table}{{host!=""}} * 2)"#
|
||||
),
|
||||
grouped_expected,
|
||||
),
|
||||
(
|
||||
format!(r#"TQL EVAL (1, 1, '1s') rate({table}{{host="h000"}}[2s:1s])"#),
|
||||
vec![(1_000, "h000".to_string(), 0.5)],
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
observer.clear();
|
||||
let rows = native_result_rows(native_tql_output(&instance, &sql, true).await).await;
|
||||
if index == 0 {
|
||||
let plan = observer.two_input_fused();
|
||||
assert_two_source_native_plan(&observer, &plan, false);
|
||||
} else {
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"a fallback shape must not plan a fused node: {sql}"
|
||||
);
|
||||
}
|
||||
assert_eq!(rows, expected_rows, "{sql}");
|
||||
let legacy_rows = native_result_rows(native_tql_output(&instance, &sql, false).await).await;
|
||||
assert_eq!(rows, legacy_rows, "{sql}");
|
||||
}
|
||||
// Group matching executes natively: it keeps the many side's series, so the route answers it
|
||||
// with one node over two same-table scans.
|
||||
let grouped_sql = format!(
|
||||
r#"TQL EVAL (0, 0, '1s') {table}{{host!=""}} - on(host) group_left() ({table}{{host!=""}} * 2)"#
|
||||
);
|
||||
observer.clear();
|
||||
let grouped_rows =
|
||||
native_result_rows(native_tql_output(&instance, &grouped_sql, true).await).await;
|
||||
let grouped_plan = observer.two_input_fused();
|
||||
assert_two_source_native_plan(&observer, &grouped_plan, false);
|
||||
assert_eq!(grouped_rows, expected[..HOSTS].to_vec(), "{grouped_sql}");
|
||||
let grouped_legacy =
|
||||
native_result_rows(native_tql_output(&instance, &grouped_sql, false).await).await;
|
||||
assert_eq!(grouped_rows, grouped_legacy, "{grouped_sql}");
|
||||
|
||||
// A subquery is planned natively: `rate` over an evaluated input reads the grid the subquery
|
||||
// produces, and the route compiles that window instead of sending the statement to the ordinary
|
||||
// planner. The rows are what says the plan it runs answers the statement it was given, so the
|
||||
// answer is pinned here and compared with the existing plan's on the same fixture.
|
||||
let subquery_sql = format!(r#"TQL EVAL (1, 1, '1s') rate({table}{{host="h000"}}[2s:1s])"#);
|
||||
observer.clear();
|
||||
let subquery_rows =
|
||||
native_result_rows(native_tql_output(&instance, &subquery_sql, true).await).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"a subquery is not a fallback shape: {subquery_sql}"
|
||||
);
|
||||
assert_eq!(
|
||||
subquery_rows,
|
||||
vec![(1_000, "h000".to_string(), 0.5)],
|
||||
"{subquery_sql}"
|
||||
);
|
||||
let subquery_legacy =
|
||||
native_result_rows(native_tql_output(&instance, &subquery_sql, false).await).await;
|
||||
assert_eq!(subquery_rows, subquery_legacy, "{subquery_sql}");
|
||||
}
|
||||
|
||||
/// Empty-key grouped matches retain the many side's labels and report kernel cardinality errors.
|
||||
@@ -8540,6 +8581,30 @@ async fn native_promql_delta_temporality_rate_and_increase_match_legacy() {
|
||||
);
|
||||
}
|
||||
|
||||
// A subquery recurses through the expression tree for the same delta-temporality guard.
|
||||
// Compare against the existing route and assert that native execution declines both functions.
|
||||
for query in [
|
||||
format!("max_over_time(rate({TABLE}[2m])[2m:1m])"),
|
||||
format!("max_over_time(increase({TABLE}[2m])[2m:1m])"),
|
||||
] {
|
||||
let sql = format!("TQL EVAL (120, 120, '1m') {query}");
|
||||
let legacy_rows = native_marker_rows(native_tql_output(&instance, &sql, false).await).await;
|
||||
assert!(
|
||||
!legacy_rows.is_empty(),
|
||||
"legacy subquery `{query}` must return rows"
|
||||
);
|
||||
observer.clear();
|
||||
let native_rows = native_marker_rows(native_tql_output(&instance, &sql, true).await).await;
|
||||
assert_eq!(
|
||||
native_rows, legacy_rows,
|
||||
"`{query}`: same rows on both routes"
|
||||
);
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"`{query}`: delta temporality inside a subquery must decline native planning"
|
||||
);
|
||||
}
|
||||
|
||||
// The control: a marker-free table (samples 10 and 30 here, unlike the table above) still fuses
|
||||
// and agrees with the existing plan, so the decline is about the marker, not the table shape.
|
||||
let control_sql = format!("TQL EVAL (120, 120, '1m') sum_over_time({CONTROL}[2m])");
|
||||
@@ -12250,3 +12315,247 @@ async fn native_promql_root_canonical_merge_keeps_the_scan_width() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A subquery evaluates its inner expression on its own grid, on the native route.
|
||||
///
|
||||
/// A window over an aggregate is the split's one admitted evaluated-input shape: the subquery's
|
||||
/// declared input grid equals the suffix's, so the split carries it rather than declining. Each
|
||||
/// answer is compared with the existing planner's, and the route is asserted through the observer.
|
||||
///
|
||||
/// A window over an evaluated input is not a per-series prefix, though: it drops the metric name, so
|
||||
/// the dist analyzer reads it as cross-series and will not place a segment over one. The shapes below
|
||||
/// it - a raw window's `rate`, a `label_replace` map - are cut instead, and where nothing below the
|
||||
/// window can carry the tree the statement keeps the plan the existing route builds. Either way it
|
||||
/// answers the rows Prometheus does, and never the "no installed planner" failure a segment the
|
||||
/// analyzer refuses produces.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_subqueries_match_legacy() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
let config = mito2::config::MitoConfig {
|
||||
experimental_series_scan_v2: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (standalone, observer) = native_test_instance(
|
||||
GreptimeDbStandaloneBuilder::new("native_promql_subqueries").with_mito_config(config),
|
||||
)
|
||||
.await;
|
||||
let instance = standalone.fe_instance().clone();
|
||||
execute_all(
|
||||
&instance,
|
||||
r#"CREATE TABLE native_subq (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_subq (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 0), ('h0', 'a', 120000, 10), ('h0', 'a', 180000, 20),
|
||||
('h1', 'a', 60000, 0), ('h1', 'a', 120000, 30), ('h1', 'a', 180000, 60),
|
||||
('h0', 'b', 60000, 0), ('h0', 'b', 120000, 40), ('h0', 'b', 180000, 80),
|
||||
('h1', 'b', 60000, 0), ('h1', 'b', 120000, 50), ('h1', 'b', 180000, 100)"#,
|
||||
QueryContext::arc(),
|
||||
)
|
||||
.await;
|
||||
|
||||
for sql in [
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time(native_subq[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(rate(native_subq[2m])[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m] @ 180)",
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(max_over_time(native_subq[2m:1m])[2m:1m])",
|
||||
// A subquery body that is a mapped selection rather than a fold: the segment under the
|
||||
// window is the `Map` that clamps, and it has to publish on the subquery's own grid - the
|
||||
// two points at 120000 and 180000 at one-minute steps - rather than on the statement's,
|
||||
// which holds the single point at 180000. The two grids therefore differ in both their
|
||||
// start and their length, not just in identity.
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time(clamp(native_subq, 0, 1)[2m:1m])",
|
||||
] {
|
||||
observer.clear();
|
||||
let mut native = native_result_rows(native_tql_output(&instance, sql, true).await).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{sql} did not execute a FusedSeriesExec: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
observer.clear();
|
||||
let mut legacy = native_result_rows(native_tql_output(&instance, sql, false).await).await;
|
||||
// The two routes emit the same rows but not in the same order: the helper sorts within a
|
||||
// batch, and a series' label is one column of that batch's own order.
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"{sql} must keep the existing plan when the native route is off"
|
||||
);
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
|
||||
// The anchored shape is compared with the existing route only at its anchor. Past the anchor the
|
||||
// existing route re-folds the subquery's window on every outer step, and the split must not copy
|
||||
// that bug, so the later steps are held to the invariant on the native answer alone: an anchored
|
||||
// subquery's range vector is fixed at 180, so each outer step has to publish 180's value for a
|
||||
// series rather than fold a fresh one.
|
||||
let anchored = "TQL EVAL (180, 360, '1m') \
|
||||
avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m] @ 180)";
|
||||
observer.clear();
|
||||
let anchored_rows = native_rows(native_tql_output(&instance, anchored, true).await, None).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{anchored} did not execute a FusedSeriesExec: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
// Every label column names the series, so `job` is part of the key as well as `host`.
|
||||
let mut series =
|
||||
std::collections::BTreeMap::<&str, std::collections::BTreeMap<i64, f64>>::new();
|
||||
for (ts, label, value) in &anchored_rows {
|
||||
assert!(
|
||||
series
|
||||
.entry(label.as_str())
|
||||
.or_default()
|
||||
.insert(*ts, *value)
|
||||
.is_none(),
|
||||
"{anchored} answered {label} twice at {ts}"
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
!series.is_empty(),
|
||||
"{anchored} answered no series: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
for step in [180000, 240000, 300000] {
|
||||
assert!(
|
||||
series.values().all(|steps| steps.contains_key(&step)),
|
||||
"{anchored} did not answer {step} for every series: {anchored_rows:?}"
|
||||
);
|
||||
}
|
||||
// An anchored subquery must not re-fold when the outer step advances.
|
||||
for (label, steps) in &series {
|
||||
let anchor = steps[&180000];
|
||||
for (step, value) in steps {
|
||||
assert_eq!(
|
||||
anchor, *value,
|
||||
"{anchored}: {label} holds the anchored value at 180000 but folded {value} at {step}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// A fold over a subquery whose input is not a bare selection. The window is not itself cut as a
|
||||
// segment - it drops the metric name, so the analyzer reads it as cross-series - and what keeps
|
||||
// the scan width is the per-series prefix below it or the canonical boundary a rewrite needs.
|
||||
// `fused` says which plan answers, and every case is compared with the existing route row by
|
||||
// row. The one shape that still keeps the existing plan is the rewrite: its window's input is
|
||||
// cross-series, so `region_local_segment` refuses the cut and there is no per-series prefix
|
||||
// inside the window to take instead.
|
||||
for (sql, label, fused) in [
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum by (job) (max_over_time(rate(native_subq[2m])[5m:1m]))",
|
||||
Some("job"),
|
||||
true,
|
||||
),
|
||||
// The rewrite writes a new label, so its map is per-series: the subquery's inner map is
|
||||
// the segment, and it publishes on the subquery's grid rather than the statement's. It
|
||||
// runs natively now; the rows below are what say it answers what the existing plan does.
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum by (x) \
|
||||
(avg_over_time(label_replace(native_subq, \"x\", \"$1\", \"job\", \"(.*)\")[5m:1m]))",
|
||||
Some("x"),
|
||||
true,
|
||||
),
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(sum by (job) \
|
||||
(max_over_time(rate(native_subq[2m])[3m:1m]))[5m:1m])",
|
||||
Some("job"),
|
||||
true,
|
||||
),
|
||||
// This shape used to keep the existing plan, on the grounds that a subquery evaluates its
|
||||
// own body on a grid of its own. It runs natively now: the `rate` over the raw scan is the
|
||||
// per-series segment and the window above it publishes, so the fused node below is the plan
|
||||
// that answers. The rows are what says that plan answers what the existing one does.
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum_over_time(rate(native_subq[5m])[1h:])",
|
||||
None,
|
||||
true,
|
||||
),
|
||||
] {
|
||||
observer.clear();
|
||||
let mut native = native_rows(native_tql_output(&instance, sql, true).await, label).await;
|
||||
assert_eq!(
|
||||
fused,
|
||||
observer.fused().is_some(),
|
||||
"{sql}: the native request must {} a FusedSeriesExec: {:#?}",
|
||||
if fused {
|
||||
"run"
|
||||
} else {
|
||||
"keep the existing plan for"
|
||||
},
|
||||
observer.root_plan()
|
||||
);
|
||||
assert!(
|
||||
!native.is_empty(),
|
||||
"{sql} answered no rows: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
let mut legacy = native_rows(native_tql_output(&instance, sql, false).await, label).await;
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
}
|
||||
|
||||
/// An aggregate inside a subquery is still an aggregate, so it keeps that operand out of the
|
||||
/// hybrid-binary join.
|
||||
///
|
||||
/// Both statements below are two single-selector vector operands on the same matching labels, which
|
||||
/// is what the join admits - but one operand folds a group one subquery down. With the fold counted
|
||||
/// as a fold, both run on the native route through `build_evaluated_segment` and answer the same
|
||||
/// rows the existing route does.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_aggregate_subquery_operands_run_natively() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
let config = mito2::config::MitoConfig {
|
||||
experimental_series_scan_v2: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (standalone, observer) = native_test_instance(
|
||||
GreptimeDbStandaloneBuilder::new("native_promql_aggregate_subquery")
|
||||
.with_mito_config(config),
|
||||
)
|
||||
.await;
|
||||
let instance = standalone.fe_instance().clone();
|
||||
execute_all(
|
||||
&instance,
|
||||
r#"CREATE TABLE native_pm (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_pm (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 0), ('h0', 'a', 120000, 10), ('h0', 'a', 180000, 20),
|
||||
('h1', 'a', 60000, 0), ('h1', 'a', 120000, 30), ('h1', 'a', 180000, 60),
|
||||
('h0', 'b', 60000, 0), ('h0', 'b', 120000, 40), ('h0', 'b', 180000, 80),
|
||||
('h1', 'b', 60000, 0), ('h1', 'b', 120000, 50), ('h1', 'b', 180000, 100);
|
||||
CREATE TABLE native_pn (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_pn (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 1), ('h0', 'a', 120000, 2), ('h0', 'a', 180000, 3),
|
||||
('h1', 'a', 60000, 4), ('h1', 'a', 120000, 5), ('h1', 'a', 180000, 6),
|
||||
('h0', 'b', 60000, 7), ('h0', 'b', 120000, 8), ('h0', 'b', 180000, 9),
|
||||
('h1', 'b', 60000, 10), ('h1', 'b', 120000, 11), ('h1', 'b', 180000, 12)"#,
|
||||
QueryContext::arc(),
|
||||
)
|
||||
.await;
|
||||
|
||||
for sql in [
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (native_pm))[5m:1m]) + on(job, host) native_pn",
|
||||
"TQL EVAL (180, 180, '1m') native_pm + on(job, host) avg_over_time((sum by (job, host) (native_pn))[5m:1m])",
|
||||
] {
|
||||
observer.clear();
|
||||
let (_, mut native) =
|
||||
native_labelled_rows(native_tql_output(&instance, sql, true).await).await;
|
||||
assert!(
|
||||
!native.is_empty(),
|
||||
"{sql} answered no rows: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{sql} must run on the native route"
|
||||
);
|
||||
observer.clear();
|
||||
let (_, mut legacy) =
|
||||
native_labelled_rows(native_tql_output(&instance, sql, false).await).await;
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user