mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 00:34:11 +00:00
feat(promql): run subqueries through the native prefix (#10085)
* feat(promql): run subqueries through the native prefix A subquery evaluates its inner expression on its own grid, so the native route now plans it instead of sending every statement containing one to the existing planner. - `collect_selectors` recurses into a subquery's expression, so its selectors are prepared like any other and each gets its scan bounds from `raw_scan_bounds`. - `calls_rate_or_increase` recurses too, so a delta-temporality table still keeps a rate or an increase inside a subquery on the existing planner. - A window whose input is already evaluated - a subquery - is admitted above the split aggregate when its declared grid equals its direct input's grid, and the suffix is rebuilt with it. A raw-sample window still cannot sit above the cut. - A `FusedSeries` whose root is such a window publishes on that window's output grid, which is not necessarily the statement's; the binding follows it at every construction site, and the split's partial state keeps the cut aggregate's own grid. An evaluated-input window does not need an owner router for its sources, and a segment whose root is one is cut on its own grid. Verified with `-p promql`, `-p query`, `tests-integration --lib` (native_promql) and clippy. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep a subquery's aggregate operand and gate an evaluated cut Two shapes reached the native route and failed while planning instead of keeping the plan they had. A hybrid-binary operand folds a group one subquery down (`avg_over_time((sum by (job, host) (pm))[5m:1m]) + on(job, host) pn`): `single_selector_operand` counts the subquery's selector, so the operand looked admissible, while `contains_aggregate_expression` stopped at `PromExpr::Subquery` and never saw the aggregate. The admission then built a `FusedSeries` the physical planner could not convert (`No installed planner was able to convert the custom node to an execution plan ...`). The walk descends into a subquery again, and both statements keep the existing plan with the same rows. A window over an evaluated input was cut as a segment regardless of locality. For any such window whose input is not a bare `Input` - a raw window, a `label_replace` - `reads_across_series` is true and `pointwise_operand` false, so the `FusedSeries` arm of the dist analyzer answers `NonCommutative`: the segment stays above the `MergeScan` and nothing converts it (`sum by (job) (max_over_time(rate(pm[2m])[5m:1m]))` and two neighbours failed that way at instant and range evaluation). `region_local_segment` mirrors that arm's decision from the tree, and `cut_as_segment` declines an evaluated-input window it refuses, cutting the per-series prefix inside the window instead; where there is none, the statement keeps the plan it had. The analyzer is not extended: every range function but `last_over_time`/`first_over_time` drops the metric name, so such a window is genuinely cross-series and a region-local segment would publish two series under one label set. `build_evaluated_segment` stays, and so does its caller. The canonical pass inserts a merge under a label rewrite, and that boundary is only ever built from the frames the loop has published, so the cut needs the fallback; deleting it took every rewritten fold off the route, which is what `sum(label_replace(...))` and the histogram-over-rewrite boundary caught. The frames-bound segment itself is not the defect - the gate is: the evaluated-input window is declined before any cut is taken, so the fallback only sees cuts the analyzer places. Tests that asserted the older rejection are decided on evidence: `sum_over_time(rate(m[5m])[1h:])` is admitted now and its rows match the existing plan, so the plan-level test asserts the route and the integration case proves the rows; the aggregate-subquery case in `native_aggregate_complete_over_four_nonempty_mito_partitions` is answered by the split (one `Partial` producer, one frontend merge) and returns the same 15.0 on both routes, so it moved to the split cases rather than being dropped. Two fold-over-subquery shapes in `native_promql_subqueries_match_legacy` (`sum by (job) (max_over_time(rate(...)[5m:1m]))` and the fold above it) are answered natively for the same reason - the fallback builds their cut - and their rows equal the existing plan's, so they assert the route instead of the fallback; the rewrite shape still declines, because its window reads across series and there is no per-series prefix inside it to take instead. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert anchored subqueries do not re-fold across steps Signed-off-by: discord9 <discord9@163.com> * test(promql): drop the unused fallback-case helper Signed-off-by: discord9 <discord9@163.com> * test(promql): update subquery expectations for the native route Signed-off-by: discord9 <discord9@163.com> * fix(promql): publish subquery segments on their declared grid Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop unreachable segment arms and pin subquery routes Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
@@ -47,12 +47,10 @@
|
||||
//! satisfy. No other counter query reads that tag: an aggregate drops the labels it does not group
|
||||
//! by, so their answers are the ones the untagged fixture gives.
|
||||
//!
|
||||
//! A query the native route declines is checked the other way round. [`check_fallback_case`] runs
|
||||
//! shapes the compiler has no node for - `scalar(v)` and a subquery - with the native request on and
|
||||
//! with it off, and requires that no native node ran in either plan and that both runs report rows
|
||||
//! derived from the fixture by hand. A hint that quietly
|
||||
//! took the route for one of them would show a fused producer or a frontend merge in the plans that
|
||||
//! executed, which is what is observed.
|
||||
//! Two shapes that used to be declined are checked on the plans that really ran. `scalar(v)` reads
|
||||
//! the whole vector in one fused node and needs no merge; a subquery over a fold splits into one
|
||||
//! producer and one frontend merge. A hint that quietly kept the existing route would show no native
|
||||
//! node, which is what the plan observer checks, and both routes must answer the hand-derived rows.
|
||||
//!
|
||||
//! A binary operation whose sibling is a literal scalar is carried by the split on either side of
|
||||
//! the fold: below the aggregate it is computed in the per-series prefix (`sum(8 * rate(...))`,
|
||||
@@ -739,9 +737,16 @@ async fn native_hybrid_binary_05_06_08() {
|
||||
);
|
||||
let (legacy, legacy_capture) = run_query(&instance, &subquery, false, &observer).await;
|
||||
let (native, native_capture) = run_query(&instance, &subquery, true, &observer).await;
|
||||
// A subquery now runs on the native route, so the existing plan must hold no native node while
|
||||
// the native run must exercise one. The rows below are what says the route answers what the
|
||||
// existing plan does.
|
||||
assert!(
|
||||
legacy_capture.fused.is_empty() && native_capture.fused.is_empty(),
|
||||
"hybrid must be disabled inside subquery: {subquery}"
|
||||
legacy_capture.fused.is_empty(),
|
||||
"the existing route must not plan native nodes: {subquery}"
|
||||
);
|
||||
assert!(
|
||||
!native_capture.fused.is_empty(),
|
||||
"the subquery must run on the native route: {subquery}"
|
||||
);
|
||||
assert_eq!(
|
||||
legacy.rows.len(),
|
||||
@@ -1606,15 +1611,14 @@ fn assert_finalize_reads_region_state(
|
||||
}
|
||||
}
|
||||
|
||||
/// Checks one query the native route declines: it must stay declined on both routes, and both must
|
||||
/// answer what the fixture derives by hand.
|
||||
/// Checks one query the native route answers by splitting the aggregate below a subquery.
|
||||
///
|
||||
/// The native request is on for one run and off for the other, so the hint being asked for and the
|
||||
/// hint being absent are both exercised: neither plan may hold a fused producer or a frontend merge,
|
||||
/// and the rows - labels, evaluation time, values and their multiplicity - are the same hand-derived
|
||||
/// ones for both. The observer is asked to have seen a plan at all, because a run that handed nothing
|
||||
/// over would otherwise satisfy "no native node" without observing anything.
|
||||
async fn check_fallback_case(
|
||||
/// The per-series prefix runs in a `Partial` producer on every partition and the window above it
|
||||
/// finishes the answer in the `Final` half over the merged values, so the native run must show
|
||||
/// exactly one producer and one frontend merge - the opposite of a fallback case - while the existing
|
||||
/// plan shows neither. Both routes must answer what the fixture derives by hand, which is what says
|
||||
/// the split's answer is the statement's answer and not an approximation of it.
|
||||
async fn check_split_case(
|
||||
instance: &Arc<Instance>,
|
||||
observer: &NativePlanObserver,
|
||||
case: &AggregateCase,
|
||||
@@ -1630,13 +1634,32 @@ async fn check_fallback_case(
|
||||
"`{query}`: the {route} run handed no plan to the observer, so nothing about the plan \
|
||||
that ran was observed"
|
||||
);
|
||||
assert!(
|
||||
capture.fused.is_empty() && capture.finalize.is_empty(),
|
||||
"`{query}`: the {route} run must keep the existing plan, but the plans that ran held {} \
|
||||
fused producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
if native {
|
||||
assert_eq!(
|
||||
1,
|
||||
capture.fused.len(),
|
||||
"`{query}`: the {route} run must split the aggregate into one producer, but the \
|
||||
plans that ran held {} producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
assert_eq!(
|
||||
1,
|
||||
capture.finalize.len(),
|
||||
"`{query}`: the {route} run must finish the fold in one frontend merge, but the \
|
||||
plans that ran held {} producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
} else {
|
||||
assert!(
|
||||
capture.fused.is_empty() && capture.finalize.is_empty(),
|
||||
"`{query}`: the {route} run must keep the existing plan, but the plans that ran \
|
||||
held {} fused producers and {} frontend merges",
|
||||
capture.fused.len(),
|
||||
capture.finalize.len()
|
||||
);
|
||||
}
|
||||
assert_rows(&result.rows, &case.expected, &format!("{route} `{query}`"));
|
||||
}
|
||||
}
|
||||
@@ -2481,20 +2504,6 @@ async fn native_aggregate_complete_over_four_nonempty_mito_partitions() {
|
||||
)
|
||||
.await;
|
||||
|
||||
// A shape the native compiler declines, checked on the plans that really ran: it may hold
|
||||
// neither a fused producer nor a frontend merge, and both routes must answer what the fixture
|
||||
// derives by hand. A subquery evaluates its own body on a grid of its own: it reads across
|
||||
// series and cannot be answered by a producer per partition, so it keeps the existing plan. The
|
||||
// subquery's window spans the two inner points `EVAL_SECS - 1` and `EVAL_SECS`, at each of
|
||||
// which the same extrapolated rate answers 15 - both samples stay further from the window's
|
||||
// ends than the 66s threshold - so the maximum is 15 as well.
|
||||
let fallback_case = case(
|
||||
format!("max_over_time((sum(rate({COUNTER_TABLE}[5m])))[2s:1s])"),
|
||||
&[(&[], 15.0)],
|
||||
false,
|
||||
);
|
||||
check_fallback_case(&instance, &observer, &fallback_case).await;
|
||||
|
||||
// `scalar` reads the whole vector and the native route answers it, so it is no longer one of
|
||||
// the declined shapes. `scalar` reports the one value present at a point only when exactly one
|
||||
// series is present at it, so the five counter series make it NaN; a route that folded one
|
||||
@@ -2540,6 +2549,20 @@ async fn native_aggregate_complete_over_four_nonempty_mito_partitions() {
|
||||
&legacy.rows,
|
||||
"native `scalar` against the existing plan",
|
||||
);
|
||||
// A subquery over a fold is not declined: the compiler binds a window over an evaluated input to
|
||||
// that input's own grid, so the split carries it - the per-series prefix runs in a `Partial`
|
||||
// producer on every partition and the window above it finishes the answer in the `Final` half
|
||||
// over the merged values. The subquery's window spans the two inner points `EVAL_SECS - 1` and
|
||||
// `EVAL_SECS`, at each of which the same extrapolated rate answers 15 - both samples stay further
|
||||
// from the window's ends than the 66s threshold - so the maximum is 15 as well.
|
||||
let split_cases = [case(
|
||||
format!("max_over_time((sum(rate({COUNTER_TABLE}[5m])))[2s:1s])"),
|
||||
&[(&[], 15.0)],
|
||||
false,
|
||||
)];
|
||||
for case in &split_cases {
|
||||
check_split_case(&instance, &observer, case).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Histogram functions above the cut merge Partial state before evaluating the complete buckets.
|
||||
|
||||
@@ -2028,7 +2028,11 @@ async fn native_site_rows(output: Output) -> Vec<(i64, String, String, f64)> {
|
||||
rows
|
||||
}
|
||||
|
||||
async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
/// The rows of a TQL answer, labelled by `label`.
|
||||
///
|
||||
/// `None` labels each row with every label column it carries, by name, so an answer whose label set
|
||||
/// is not the `host` one can be compared with the existing route's the same way.
|
||||
async fn native_rows(output: Output, label: Option<&str>) -> Vec<(i64, String, f64)> {
|
||||
let batches = match output.data {
|
||||
OutputData::Stream(stream) => collect_batches(stream).await.unwrap(),
|
||||
OutputData::RecordBatches(batches) => batches,
|
||||
@@ -2043,30 +2047,60 @@ async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
.as_any()
|
||||
.downcast_ref::<TimestampMillisecondArray>()
|
||||
.unwrap();
|
||||
let hosts = cast(
|
||||
batch.column_by_name("host").unwrap().as_ref(),
|
||||
&DataType::Utf8,
|
||||
)
|
||||
.unwrap();
|
||||
let hosts = hosts.as_any().downcast_ref::<StringArray>().unwrap();
|
||||
let value_columns = batch
|
||||
.columns()
|
||||
.iter()
|
||||
.filter(|column| column.data_type() == &DataType::Float64)
|
||||
.enumerate()
|
||||
.filter(|(_, column)| column.data_type() == &DataType::Float64)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(value_columns.len(), 1, "expected one Float64 value column");
|
||||
let values = value_columns[0]
|
||||
let (value_index, value_column) = value_columns[0];
|
||||
let labels = match label {
|
||||
Some(label) => {
|
||||
let labels = cast(
|
||||
batch.column_by_name(label).unwrap().as_ref(),
|
||||
&DataType::Utf8,
|
||||
)
|
||||
.unwrap();
|
||||
let labels = labels.as_any().downcast_ref::<StringArray>().unwrap();
|
||||
assert_eq!(labels.null_count(), 0);
|
||||
(0..batch.num_rows())
|
||||
.map(|row| labels.value(row).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
}
|
||||
// Every label column, by name: the value column and the time index are the two that
|
||||
// are not one, and a route names its value differently from the other.
|
||||
None => (0..batch.num_rows())
|
||||
.map(|row| {
|
||||
batch
|
||||
.schema
|
||||
.column_schemas()
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(index, column)| *index != value_index && column.name != "ts")
|
||||
.filter_map(|(_, column)| {
|
||||
let values = batch.column_by_name(&column.name)?.as_ref();
|
||||
let values = cast(values, &DataType::Utf8).ok()?;
|
||||
let values = values.as_any().downcast_ref::<StringArray>()?;
|
||||
(!values.is_null(row))
|
||||
.then(|| format!("{}={}", column.name, values.value(row)))
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join(",")
|
||||
})
|
||||
.collect::<Vec<_>>(),
|
||||
};
|
||||
let values = value_column
|
||||
.as_any()
|
||||
.downcast_ref::<Float64Array>()
|
||||
.unwrap();
|
||||
assert_eq!(timestamps.null_count(), 0);
|
||||
assert_eq!(hosts.null_count(), 0);
|
||||
assert_eq!(values.null_count(), 0);
|
||||
(0..batch.num_rows())
|
||||
.map(|row| {
|
||||
(
|
||||
timestamps.value(row),
|
||||
hosts.value(row).to_string(),
|
||||
labels[row].clone(),
|
||||
values.value(row),
|
||||
)
|
||||
})
|
||||
@@ -2077,6 +2111,10 @@ async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
rows
|
||||
}
|
||||
|
||||
async fn native_result_rows(output: Output) -> Vec<(i64, String, f64)> {
|
||||
native_rows(output, Some("host")).await
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_clamp_aggregates_execute_on_native_segments() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
@@ -7248,38 +7286,41 @@ async fn native_promql_repeated_selector_binary_uses_independent_scans() {
|
||||
native_result_rows(native_tql_output(&instance, &different_matchers, false).await).await;
|
||||
assert_eq!(different_matcher_rows, different_matcher_legacy);
|
||||
|
||||
// Group matching executes natively; the subquery case remains on the ordinary path, unchanged.
|
||||
let grouped_expected = expected[..HOSTS].to_vec();
|
||||
for (index, (sql, expected_rows)) in [
|
||||
(
|
||||
format!(
|
||||
r#"TQL EVAL (0, 0, '1s') {table}{{host!=""}} - on(host) group_left() ({table}{{host!=""}} * 2)"#
|
||||
),
|
||||
grouped_expected,
|
||||
),
|
||||
(
|
||||
format!(r#"TQL EVAL (1, 1, '1s') rate({table}{{host="h000"}}[2s:1s])"#),
|
||||
vec![(1_000, "h000".to_string(), 0.5)],
|
||||
),
|
||||
]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
observer.clear();
|
||||
let rows = native_result_rows(native_tql_output(&instance, &sql, true).await).await;
|
||||
if index == 0 {
|
||||
let plan = observer.two_input_fused();
|
||||
assert_two_source_native_plan(&observer, &plan, false);
|
||||
} else {
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"a fallback shape must not plan a fused node: {sql}"
|
||||
);
|
||||
}
|
||||
assert_eq!(rows, expected_rows, "{sql}");
|
||||
let legacy_rows = native_result_rows(native_tql_output(&instance, &sql, false).await).await;
|
||||
assert_eq!(rows, legacy_rows, "{sql}");
|
||||
}
|
||||
// Group matching executes natively: it keeps the many side's series, so the route answers it
|
||||
// with one node over two same-table scans.
|
||||
let grouped_sql = format!(
|
||||
r#"TQL EVAL (0, 0, '1s') {table}{{host!=""}} - on(host) group_left() ({table}{{host!=""}} * 2)"#
|
||||
);
|
||||
observer.clear();
|
||||
let grouped_rows =
|
||||
native_result_rows(native_tql_output(&instance, &grouped_sql, true).await).await;
|
||||
let grouped_plan = observer.two_input_fused();
|
||||
assert_two_source_native_plan(&observer, &grouped_plan, false);
|
||||
assert_eq!(grouped_rows, expected[..HOSTS].to_vec(), "{grouped_sql}");
|
||||
let grouped_legacy =
|
||||
native_result_rows(native_tql_output(&instance, &grouped_sql, false).await).await;
|
||||
assert_eq!(grouped_rows, grouped_legacy, "{grouped_sql}");
|
||||
|
||||
// A subquery is planned natively: `rate` over an evaluated input reads the grid the subquery
|
||||
// produces, and the route compiles that window instead of sending the statement to the ordinary
|
||||
// planner. The rows are what says the plan it runs answers the statement it was given, so the
|
||||
// answer is pinned here and compared with the existing plan's on the same fixture.
|
||||
let subquery_sql = format!(r#"TQL EVAL (1, 1, '1s') rate({table}{{host="h000"}}[2s:1s])"#);
|
||||
observer.clear();
|
||||
let subquery_rows =
|
||||
native_result_rows(native_tql_output(&instance, &subquery_sql, true).await).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"a subquery is not a fallback shape: {subquery_sql}"
|
||||
);
|
||||
assert_eq!(
|
||||
subquery_rows,
|
||||
vec![(1_000, "h000".to_string(), 0.5)],
|
||||
"{subquery_sql}"
|
||||
);
|
||||
let subquery_legacy =
|
||||
native_result_rows(native_tql_output(&instance, &subquery_sql, false).await).await;
|
||||
assert_eq!(subquery_rows, subquery_legacy, "{subquery_sql}");
|
||||
}
|
||||
|
||||
/// Empty-key grouped matches retain the many side's labels and report kernel cardinality errors.
|
||||
@@ -8540,6 +8581,30 @@ async fn native_promql_delta_temporality_rate_and_increase_match_legacy() {
|
||||
);
|
||||
}
|
||||
|
||||
// A subquery recurses through the expression tree for the same delta-temporality guard.
|
||||
// Compare against the existing route and assert that native execution declines both functions.
|
||||
for query in [
|
||||
format!("max_over_time(rate({TABLE}[2m])[2m:1m])"),
|
||||
format!("max_over_time(increase({TABLE}[2m])[2m:1m])"),
|
||||
] {
|
||||
let sql = format!("TQL EVAL (120, 120, '1m') {query}");
|
||||
let legacy_rows = native_marker_rows(native_tql_output(&instance, &sql, false).await).await;
|
||||
assert!(
|
||||
!legacy_rows.is_empty(),
|
||||
"legacy subquery `{query}` must return rows"
|
||||
);
|
||||
observer.clear();
|
||||
let native_rows = native_marker_rows(native_tql_output(&instance, &sql, true).await).await;
|
||||
assert_eq!(
|
||||
native_rows, legacy_rows,
|
||||
"`{query}`: same rows on both routes"
|
||||
);
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"`{query}`: delta temporality inside a subquery must decline native planning"
|
||||
);
|
||||
}
|
||||
|
||||
// The control: a marker-free table (samples 10 and 30 here, unlike the table above) still fuses
|
||||
// and agrees with the existing plan, so the decline is about the marker, not the table shape.
|
||||
let control_sql = format!("TQL EVAL (120, 120, '1m') sum_over_time({CONTROL}[2m])");
|
||||
@@ -12250,3 +12315,247 @@ async fn native_promql_root_canonical_merge_keeps_the_scan_width() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A subquery evaluates its inner expression on its own grid, on the native route.
|
||||
///
|
||||
/// A window over an aggregate is the split's one admitted evaluated-input shape: the subquery's
|
||||
/// declared input grid equals the suffix's, so the split carries it rather than declining. Each
|
||||
/// answer is compared with the existing planner's, and the route is asserted through the observer.
|
||||
///
|
||||
/// A window over an evaluated input is not a per-series prefix, though: it drops the metric name, so
|
||||
/// the dist analyzer reads it as cross-series and will not place a segment over one. The shapes below
|
||||
/// it - a raw window's `rate`, a `label_replace` map - are cut instead, and where nothing below the
|
||||
/// window can carry the tree the statement keeps the plan the existing route builds. Either way it
|
||||
/// answers the rows Prometheus does, and never the "no installed planner" failure a segment the
|
||||
/// analyzer refuses produces.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_subqueries_match_legacy() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
let config = mito2::config::MitoConfig {
|
||||
experimental_series_scan_v2: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (standalone, observer) = native_test_instance(
|
||||
GreptimeDbStandaloneBuilder::new("native_promql_subqueries").with_mito_config(config),
|
||||
)
|
||||
.await;
|
||||
let instance = standalone.fe_instance().clone();
|
||||
execute_all(
|
||||
&instance,
|
||||
r#"CREATE TABLE native_subq (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_subq (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 0), ('h0', 'a', 120000, 10), ('h0', 'a', 180000, 20),
|
||||
('h1', 'a', 60000, 0), ('h1', 'a', 120000, 30), ('h1', 'a', 180000, 60),
|
||||
('h0', 'b', 60000, 0), ('h0', 'b', 120000, 40), ('h0', 'b', 180000, 80),
|
||||
('h1', 'b', 60000, 0), ('h1', 'b', 120000, 50), ('h1', 'b', 180000, 100)"#,
|
||||
QueryContext::arc(),
|
||||
)
|
||||
.await;
|
||||
|
||||
for sql in [
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time(native_subq[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(rate(native_subq[2m])[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m])",
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m] @ 180)",
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(max_over_time(native_subq[2m:1m])[2m:1m])",
|
||||
// A subquery body that is a mapped selection rather than a fold: the segment under the
|
||||
// window is the `Map` that clamps, and it has to publish on the subquery's own grid - the
|
||||
// two points at 120000 and 180000 at one-minute steps - rather than on the statement's,
|
||||
// which holds the single point at 180000. The two grids therefore differ in both their
|
||||
// start and their length, not just in identity.
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time(clamp(native_subq, 0, 1)[2m:1m])",
|
||||
] {
|
||||
observer.clear();
|
||||
let mut native = native_result_rows(native_tql_output(&instance, sql, true).await).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{sql} did not execute a FusedSeriesExec: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
observer.clear();
|
||||
let mut legacy = native_result_rows(native_tql_output(&instance, sql, false).await).await;
|
||||
// The two routes emit the same rows but not in the same order: the helper sorts within a
|
||||
// batch, and a series' label is one column of that batch's own order.
|
||||
assert!(
|
||||
observer.fused().is_none(),
|
||||
"{sql} must keep the existing plan when the native route is off"
|
||||
);
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
|
||||
// The anchored shape is compared with the existing route only at its anchor. Past the anchor the
|
||||
// existing route re-folds the subquery's window on every outer step, and the split must not copy
|
||||
// that bug, so the later steps are held to the invariant on the native answer alone: an anchored
|
||||
// subquery's range vector is fixed at 180, so each outer step has to publish 180's value for a
|
||||
// series rather than fold a fresh one.
|
||||
let anchored = "TQL EVAL (180, 360, '1m') \
|
||||
avg_over_time((sum by (job, host) (rate(native_subq[2m])))[2m:1m] @ 180)";
|
||||
observer.clear();
|
||||
let anchored_rows = native_rows(native_tql_output(&instance, anchored, true).await, None).await;
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{anchored} did not execute a FusedSeriesExec: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
// Every label column names the series, so `job` is part of the key as well as `host`.
|
||||
let mut series =
|
||||
std::collections::BTreeMap::<&str, std::collections::BTreeMap<i64, f64>>::new();
|
||||
for (ts, label, value) in &anchored_rows {
|
||||
assert!(
|
||||
series
|
||||
.entry(label.as_str())
|
||||
.or_default()
|
||||
.insert(*ts, *value)
|
||||
.is_none(),
|
||||
"{anchored} answered {label} twice at {ts}"
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
!series.is_empty(),
|
||||
"{anchored} answered no series: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
for step in [180000, 240000, 300000] {
|
||||
assert!(
|
||||
series.values().all(|steps| steps.contains_key(&step)),
|
||||
"{anchored} did not answer {step} for every series: {anchored_rows:?}"
|
||||
);
|
||||
}
|
||||
// An anchored subquery must not re-fold when the outer step advances.
|
||||
for (label, steps) in &series {
|
||||
let anchor = steps[&180000];
|
||||
for (step, value) in steps {
|
||||
assert_eq!(
|
||||
anchor, *value,
|
||||
"{anchored}: {label} holds the anchored value at 180000 but folded {value} at {step}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// A fold over a subquery whose input is not a bare selection. The window is not itself cut as a
|
||||
// segment - it drops the metric name, so the analyzer reads it as cross-series - and what keeps
|
||||
// the scan width is the per-series prefix below it or the canonical boundary a rewrite needs.
|
||||
// `fused` says which plan answers, and every case is compared with the existing route row by
|
||||
// row. The one shape that still keeps the existing plan is the rewrite: its window's input is
|
||||
// cross-series, so `region_local_segment` refuses the cut and there is no per-series prefix
|
||||
// inside the window to take instead.
|
||||
for (sql, label, fused) in [
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum by (job) (max_over_time(rate(native_subq[2m])[5m:1m]))",
|
||||
Some("job"),
|
||||
true,
|
||||
),
|
||||
// The rewrite writes a new label, so its map is per-series: the subquery's inner map is
|
||||
// the segment, and it publishes on the subquery's grid rather than the statement's. It
|
||||
// runs natively now; the rows below are what say it answers what the existing plan does.
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum by (x) \
|
||||
(avg_over_time(label_replace(native_subq, \"x\", \"$1\", \"job\", \"(.*)\")[5m:1m]))",
|
||||
Some("x"),
|
||||
true,
|
||||
),
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') max_over_time(sum by (job) \
|
||||
(max_over_time(rate(native_subq[2m])[3m:1m]))[5m:1m])",
|
||||
Some("job"),
|
||||
true,
|
||||
),
|
||||
// This shape used to keep the existing plan, on the grounds that a subquery evaluates its
|
||||
// own body on a grid of its own. It runs natively now: the `rate` over the raw scan is the
|
||||
// per-series segment and the window above it publishes, so the fused node below is the plan
|
||||
// that answers. The rows are what says that plan answers what the existing one does.
|
||||
(
|
||||
"TQL EVAL (180, 180, '1m') sum_over_time(rate(native_subq[5m])[1h:])",
|
||||
None,
|
||||
true,
|
||||
),
|
||||
] {
|
||||
observer.clear();
|
||||
let mut native = native_rows(native_tql_output(&instance, sql, true).await, label).await;
|
||||
assert_eq!(
|
||||
fused,
|
||||
observer.fused().is_some(),
|
||||
"{sql}: the native request must {} a FusedSeriesExec: {:#?}",
|
||||
if fused {
|
||||
"run"
|
||||
} else {
|
||||
"keep the existing plan for"
|
||||
},
|
||||
observer.root_plan()
|
||||
);
|
||||
assert!(
|
||||
!native.is_empty(),
|
||||
"{sql} answered no rows: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
let mut legacy = native_rows(native_tql_output(&instance, sql, false).await, label).await;
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
}
|
||||
|
||||
/// An aggregate inside a subquery is still an aggregate, so it keeps that operand out of the
|
||||
/// hybrid-binary join.
|
||||
///
|
||||
/// Both statements below are two single-selector vector operands on the same matching labels, which
|
||||
/// is what the join admits - but one operand folds a group one subquery down. With the fold counted
|
||||
/// as a fold, both run on the native route through `build_evaluated_segment` and answer the same
|
||||
/// rows the existing route does.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn native_promql_aggregate_subquery_operands_run_natively() {
|
||||
common_telemetry::init_default_ut_logging();
|
||||
let config = mito2::config::MitoConfig {
|
||||
experimental_series_scan_v2: true,
|
||||
..Default::default()
|
||||
};
|
||||
let (standalone, observer) = native_test_instance(
|
||||
GreptimeDbStandaloneBuilder::new("native_promql_aggregate_subquery")
|
||||
.with_mito_config(config),
|
||||
)
|
||||
.await;
|
||||
let instance = standalone.fe_instance().clone();
|
||||
execute_all(
|
||||
&instance,
|
||||
r#"CREATE TABLE native_pm (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_pm (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 0), ('h0', 'a', 120000, 10), ('h0', 'a', 180000, 20),
|
||||
('h1', 'a', 60000, 0), ('h1', 'a', 120000, 30), ('h1', 'a', 180000, 60),
|
||||
('h0', 'b', 60000, 0), ('h0', 'b', 120000, 40), ('h0', 'b', 180000, 80),
|
||||
('h1', 'b', 60000, 0), ('h1', 'b', 120000, 50), ('h1', 'b', 180000, 100);
|
||||
CREATE TABLE native_pn (host STRING, job STRING, ts TIMESTAMP(3) TIME INDEX, val DOUBLE, PRIMARY KEY (host, job));
|
||||
INSERT INTO native_pn (host, job, ts, val) VALUES
|
||||
('h0', 'a', 60000, 1), ('h0', 'a', 120000, 2), ('h0', 'a', 180000, 3),
|
||||
('h1', 'a', 60000, 4), ('h1', 'a', 120000, 5), ('h1', 'a', 180000, 6),
|
||||
('h0', 'b', 60000, 7), ('h0', 'b', 120000, 8), ('h0', 'b', 180000, 9),
|
||||
('h1', 'b', 60000, 10), ('h1', 'b', 120000, 11), ('h1', 'b', 180000, 12)"#,
|
||||
QueryContext::arc(),
|
||||
)
|
||||
.await;
|
||||
|
||||
for sql in [
|
||||
"TQL EVAL (180, 180, '1m') avg_over_time((sum by (job, host) (native_pm))[5m:1m]) + on(job, host) native_pn",
|
||||
"TQL EVAL (180, 180, '1m') native_pm + on(job, host) avg_over_time((sum by (job, host) (native_pn))[5m:1m])",
|
||||
] {
|
||||
observer.clear();
|
||||
let (_, mut native) =
|
||||
native_labelled_rows(native_tql_output(&instance, sql, true).await).await;
|
||||
assert!(
|
||||
!native.is_empty(),
|
||||
"{sql} answered no rows: {:#?}",
|
||||
observer.root_plan()
|
||||
);
|
||||
assert!(
|
||||
observer.fused().is_some(),
|
||||
"{sql} must run on the native route"
|
||||
);
|
||||
observer.clear();
|
||||
let (_, mut legacy) =
|
||||
native_labelled_rows(native_tql_output(&instance, sql, false).await).await;
|
||||
native.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
legacy.sort_by(|a, b| (a.0, &a.1).cmp(&(b.0, &b.1)).then(a.2.total_cmp(&b.2)));
|
||||
assert_eq!(native, legacy, "{sql}");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user