mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
feat(promql): compile a constant, a negation and a scalar-vector operation
Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's
grid, a negation binds as the multiplication by `-1` it is, and a binary operation
between one vector and one constant binds as a `Binary` node whose output is the
vector side's own binding. The arithmetic and comparison operators all map, and
`return_bool` is carried as its own field so `x > 0` (which filters) and
`x > bool 0` (which reports zero or one) are one node with one field changed.
The operands keep the order they were written in, because most operators are not
commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only
thing that tells them apart. A constant is bound by the same `constant_value` fold a
call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition
the tree would have to know how to run.
Three shapes are refused rather than approximated. An operation between two vectors
needs each `Input` to name the selector it reads, which the one binding does not
carry, so `a + b` and `a + a` would compile to the same tree; a set operator
combines two vectors' series and has no node at all; and a statement whose value is
a scalar has nothing to scan, so a constant is only bound next to a vector that
gives it something to broadcast over. Each refusal names what it found.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit e672a9c526)
This commit is contained in:
@@ -23,15 +23,20 @@
|
||||
//! <range function>(<selector>[<range>]) e.g. rate(x[5m])
|
||||
//! <range function>(<constant>, <selector>[<range>]) e.g. quantile_over_time(0.9, x[5m])
|
||||
//! <math function>(<expression>[, <constant>]*) e.g. abs(rate(x[5m])), clamp_min(x, 0)
|
||||
//! <expression> <operator> <constant> e.g. x * 100, x > bool 0
|
||||
//! <constant> <operator> <expression> e.g. 1 / x, 0 < x
|
||||
//! -<expression> e.g. -x
|
||||
//! ```
|
||||
//!
|
||||
//! which become `Input` (raw samples) → `Instant` (the last sample within the statement's lookback),
|
||||
//! `Input` (raw samples) → `Window` (the range function over the statement's grid) and
|
||||
//! `<expression>` → `Map` (a pointwise math function over what its argument evaluates to). A
|
||||
//! selector may carry `offset` in either selection shape and, in the window shape, an `@` anchor.
|
||||
//! Everything else is a [`DataFusionError::NotImplemented`] that names what was found and quotes the
|
||||
//! shapes above, because rejecting a query the compiler does not understand is the only honest
|
||||
//! alternative to compiling it into something else.
|
||||
//! `Input` (raw samples) → `Window` (the range function over the statement's grid),
|
||||
//! `<expression>` → `Map` (a pointwise math function over what its argument evaluates to) and, over
|
||||
//! any of those, `Binary` (an arithmetic or comparison operation with a constant on one side) and
|
||||
//! `Scalar` (that constant). A selector may carry
|
||||
//! `offset` in either selection shape and, in the window shape, an `@` anchor. Everything else is a
|
||||
//! [`DataFusionError::NotImplemented`] that names what was found and quotes the shapes above,
|
||||
//! because rejecting a query the compiler does not understand is the only honest alternative to
|
||||
//! compiling it into something else.
|
||||
//!
|
||||
//! Two decisions keep the recursion from growing a branch per query. A constant argument — a
|
||||
//! quantile, a rounding step, a clamp bound — is folded out of the statement by [`constant_value`],
|
||||
@@ -42,6 +47,20 @@
|
||||
//! alone, so the number this compiler binds and the number the kernel computes cannot disagree
|
||||
//! about `%`, `^` or `atan2`.
|
||||
//!
|
||||
//! Two more are about what a node means rather than about which node to build. A binary operation
|
||||
//! keeps its operands in the order they were written, because most operators are not commutative:
|
||||
//! `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them
|
||||
//! apart. `return_bool` is carried as its own field, so `x > 0` (which filters) and `x > bool 0`
|
||||
//! (which reports zero or one) are one node with one field changed. A negation then needs no node of
|
||||
//! its own: `-x` binds as the multiplication by `-1` that it is, which the multiplication kernel
|
||||
//! already computes.
|
||||
//!
|
||||
//! What a tree cannot yet express is deliberate too. The one input binding carries the ordering it
|
||||
//! guarantees and what it produces, but not the selector it reads, so a tree with two selectors
|
||||
//! cannot tell its inputs apart and an operation between two vectors is refused. A statement whose
|
||||
//! value is a scalar has nothing to read, so it is refused as well: a constant is bound next to a
|
||||
//! vector that gives it something to broadcast over.
|
||||
//!
|
||||
//! The statement's grid, the selector's lookback, window, offset and anchor are the only numbers
|
||||
//! derived here; the legality of the tree is not checked here. [`NativeTree::try_new`] is the
|
||||
//! single authority on that, so a node added later cannot be legal in one place and illegal in
|
||||
@@ -58,14 +77,16 @@ use datatypes::series::samples::SeriesSchema;
|
||||
use datatypes::series::time::{EvalGrid, GridSpec};
|
||||
use promql::map::{ArithmeticOp, MathOp};
|
||||
use promql::native::plan::{
|
||||
BoundInput, BoundInstant, BoundMap, BoundWindow, NativeExpr, NativeNode, NativeTree, NativeType,
|
||||
BoundInput, BoundInstant, BoundMap, BoundScalar, BoundWindow, NativeExpr, NativeNode,
|
||||
NativeTree, NativeType,
|
||||
};
|
||||
use promql::range_eval::{RangeFunction, RangeWindow};
|
||||
use promql::vector_match::{BinaryMatching, BinaryOp, Cardinality, FillValues, MatchingLabels};
|
||||
use promql_parser::parser::token::TokenType;
|
||||
use promql_parser::parser::value::ValueType;
|
||||
use promql_parser::parser::{
|
||||
AtModifier, Call, EvalStmt, Expr as PromExpr, MatrixSelector, NumberLiteral, Offset,
|
||||
VectorSelector, token,
|
||||
AtModifier, BinaryExpr, Call, EvalStmt, Expr as PromExpr, MatrixSelector, NumberLiteral,
|
||||
Offset, VectorMatchCardinality, VectorMatchFillValues, VectorSelector, token,
|
||||
};
|
||||
|
||||
use crate::promql::label_values::signed_millis_since_epoch;
|
||||
@@ -73,9 +94,11 @@ use crate::promql::label_values::signed_millis_since_epoch;
|
||||
/// The shapes this compiler accepts, quoted in every rejection.
|
||||
const SUPPORTED: &str = "a selector, a range function over a range selector — \
|
||||
`<range function>(<selector>[<range>])`, with its constants in front of \
|
||||
the selector as in `quantile_over_time(0.9, x[5m])` — or a math function \
|
||||
over any of them, as in `abs(rate(x[5m]))`: for example `x`, \
|
||||
`rate(x[5m])`, `quantile_over_time(0.9, x[5m])` or `abs(rate(x[5m]))`";
|
||||
the selector as in `quantile_over_time(0.9, x[5m])` — a math function over \
|
||||
any of them, a binary operation between one of them and a constant, or a \
|
||||
negation of one of them, as in `x`, `rate(x[5m])`, \
|
||||
`quantile_over_time(0.9, x[5m])`, `abs(rate(x[5m]))`, `2 / (x + 1)` or \
|
||||
`-x`";
|
||||
|
||||
/// Compiles `stmt` into a tree that reads `input` and evaluates the statement's expression over it.
|
||||
///
|
||||
@@ -114,6 +137,18 @@ pub fn compile(
|
||||
series may not interleave",
|
||||
));
|
||||
}
|
||||
// A tree reads a scan and reports evaluated samples, so an expression whose value is not a
|
||||
// vector has nothing to compile into: `1 + 2` is a scalar, and the scalar node is bound as a
|
||||
// constant next to a vector that gives it something to broadcast over rather than on its own.
|
||||
// The check is on the statement's type rather than on the compiled node, so it does not depend on
|
||||
// which kinds of expression happen to compile to a scalar today.
|
||||
if stmt.expr.value_type() != ValueType::Vector {
|
||||
return Err(not_implemented(&format!(
|
||||
"the expression `{}`, which is a {} where a tree reports evaluated samples",
|
||||
stmt.expr,
|
||||
stmt.expr.value_type()
|
||||
)));
|
||||
}
|
||||
let context = Context {
|
||||
stmt,
|
||||
input,
|
||||
@@ -145,16 +180,190 @@ struct Context<'a> {
|
||||
/// its argument, so a query nests as deeply as it was written. A kind this compiler has no node for
|
||||
/// is refused with what was found, which is why the rejection is useful without a coverage list.
|
||||
fn compile_expr(context: &Context, expr: &PromExpr) -> Result<NativeNode> {
|
||||
// A constant is a value rather than a computation, so it binds as the number it folds to, by the
|
||||
// same fold a call's parameters use: `x * (1 + 2)` carries `3` rather than an addition the tree
|
||||
// would have to know how to run. The fold is checked first so that no shape below has to say
|
||||
// what it does with a constant operand.
|
||||
if let Some(value) = constant_value(expr) {
|
||||
return scalar_node(context, value);
|
||||
}
|
||||
match expr {
|
||||
// Parentheses group, so they change how a statement was typed rather than what it asks for:
|
||||
// the expression inside them is the one that compiles.
|
||||
PromExpr::Paren(paren) => compile_expr(context, &paren.expr),
|
||||
PromExpr::Call(call) => compile_call(context, call),
|
||||
PromExpr::VectorSelector(selector) => instant_selection(context, selector),
|
||||
PromExpr::Unary(unary) => negate(context, &unary.expr),
|
||||
PromExpr::Binary(binary) => binary_operation(context, binary),
|
||||
found => Err(not_implemented(&format!("the expression `{found}`"))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Binds `value` as a constant scalar on the statement's grid.
|
||||
///
|
||||
/// The grid is the statement's rather than the constant's own: a constant has no timestamps, so the
|
||||
/// only positions it can be broadcast over are the ones the statement is evaluated at, and
|
||||
/// [`NativeTree::validate`] compares a binary operation's two grids by spec rather than by
|
||||
/// allocation.
|
||||
fn scalar_node(context: &Context, value: f64) -> Result<NativeNode> {
|
||||
Ok(NativeNode {
|
||||
expr: NativeExpr::Scalar(BoundScalar { value }),
|
||||
output: NativeType::Scalar(statement_grid(context.stmt)?),
|
||||
})
|
||||
}
|
||||
|
||||
/// Compiles a negation as the multiplication it is: `-x` is `-1 * x`.
|
||||
///
|
||||
/// The tree has no negate node and needs none: broadcasting a constant over the expression's value
|
||||
/// is the same arithmetic, and the multiplication kernel is the one that already computes it. A
|
||||
/// dedicated negate kernel would be an optimization of this node's work rather than a different
|
||||
/// meaning, so binding the multiplication does not bind anything the query did not ask for.
|
||||
fn negate(context: &Context, operand: &PromExpr) -> Result<NativeNode> {
|
||||
// The constant goes on the left, where the minus was: `-x` subtracts nothing, but a reader of
|
||||
// the tree should see the same order the query was written in.
|
||||
Ok(broadcast(
|
||||
scalar_node(context, -1.0)?,
|
||||
compile_expr(context, operand)?,
|
||||
BinaryOp::Mul,
|
||||
false,
|
||||
))
|
||||
}
|
||||
|
||||
/// Compiles a binary operation whose operands are not both vectors.
|
||||
///
|
||||
/// The tree holds one input binding, and that binding carries the ordering it guarantees and what it
|
||||
/// produces but not the selector it reads, so two vectors cannot be told apart: `a + b` and `a + a`
|
||||
/// would compile to the same tree. That is why a vector–vector operation is refused until an input
|
||||
/// names its selector, and why an operation over one vector and one scalar is the shape the tree can
|
||||
/// bind: a scalar carries no labels, so broadcasting it is unambiguous.
|
||||
fn binary_operation(context: &Context, binary: &BinaryExpr) -> Result<NativeNode> {
|
||||
// A set operator combines two vectors' series rather than their values, and the tree has no node
|
||||
// for it at all. Its name is reported before the operands are looked at, so a set operation
|
||||
// between two vectors is refused as the set operation it is rather than as a binary operation
|
||||
// the compiler has no node for.
|
||||
if binary.op.is_set_operator() {
|
||||
return Err(not_implemented(&format!(
|
||||
"the set operator `{}` in `{binary}`, which has no native node: it combines two \
|
||||
vectors' series, and every binary node here is an arithmetic or comparison operation \
|
||||
over values",
|
||||
binary.op
|
||||
)));
|
||||
}
|
||||
let Some(op) = binary_op(&binary.op) else {
|
||||
return Err(not_implemented(&format!(
|
||||
"the binary operator `{}` in `{binary}`",
|
||||
binary.op
|
||||
)));
|
||||
};
|
||||
// Two vectors are refused before the modifier is looked at, so a vector–vector query with
|
||||
// matching says what is actually missing. The refusal names the missing identity rather than the
|
||||
// query, because that is what whoever adds it has to add.
|
||||
if binary.lhs.value_type() == ValueType::Vector && binary.rhs.value_type() == ValueType::Vector
|
||||
{
|
||||
return Err(not_implemented(&format!(
|
||||
"the binary operation `{binary}` between two vectors, which needs each input to name \
|
||||
the selector it reads: an input binding carries its ordering and what it produces, but \
|
||||
not which selector it was bound to"
|
||||
)));
|
||||
}
|
||||
// What is left has a constant on at least one side, and a modifier is refused rather than
|
||||
// dropped: a scalar carries no labels, so there is nothing to match or name, and Prometheus
|
||||
// rejects these words on a scalar operand for the same reason. A fill value is refused because
|
||||
// the tree's matching has no field for one, so binding the operation would compute it without
|
||||
// the fill the query asked for.
|
||||
if let Some(modifier) = &binary.modifier
|
||||
&& (modifier.matching.is_some()
|
||||
|| modifier.card != VectorMatchCardinality::OneToOne
|
||||
|| modifier.fill_values != VectorMatchFillValues::default())
|
||||
{
|
||||
return Err(not_implemented(&format!(
|
||||
"the vector matching in `{binary}`, which a constant operand has nothing to match: \
|
||||
`on`, `ignoring`, `group_left` and `group_right` name labels of a vector, and a fill \
|
||||
value fills in a match group that could miss"
|
||||
)));
|
||||
}
|
||||
// Both operands are compiled before the node is built, so `2 / (x + 1)` nests. The constant side
|
||||
// is bound by [`compile_expr`]'s fold, which is what makes it a scalar node rather than an
|
||||
// expression the tree would have to run.
|
||||
Ok(broadcast(
|
||||
compile_expr(context, &binary.lhs)?,
|
||||
compile_expr(context, &binary.rhs)?,
|
||||
op,
|
||||
binary.return_bool(),
|
||||
))
|
||||
}
|
||||
|
||||
/// Wraps two compiled operands in the binary node that applies `op` to them.
|
||||
///
|
||||
/// The output is the vector side's own binding, because a scalar carries no fields: a broadcast keeps
|
||||
/// the vector's fields and the grid it is evaluated on, which is the statement's grid on both sides.
|
||||
/// Two scalars are possible as a nested operand — a pair that [`constant_value`] refuses to fold, such
|
||||
/// as a comparison between two constants — and the node reports the left one's grid, which is the
|
||||
/// statement's too. The operands keep the order they were written in, because a non-commutative
|
||||
/// operator means what that order says.
|
||||
fn broadcast(left: NativeNode, right: NativeNode, op: BinaryOp, return_bool: bool) -> NativeNode {
|
||||
// `compile_expr` binds a constant to a scalar node and a selection or a function to a vector, so
|
||||
// this is where the side the operation reports is decided.
|
||||
let output = if matches!(left.output, NativeType::Scalar(_)) {
|
||||
right.output.clone()
|
||||
} else {
|
||||
left.output.clone()
|
||||
};
|
||||
NativeNode {
|
||||
expr: NativeExpr::Binary {
|
||||
left: Box::new(left),
|
||||
right: Box::new(right),
|
||||
matching: scalar_matching(op, return_bool),
|
||||
},
|
||||
output,
|
||||
}
|
||||
}
|
||||
|
||||
/// The matching a vector-and-constant operation records.
|
||||
///
|
||||
/// Nothing is matched: a scalar carries no labels, so [`NativeTree::validate`] broadcasts it and
|
||||
/// reads neither `labels` nor `cardinality`. They carry the default a vector–vector operation of the
|
||||
/// same operator would have, which is also the only matching PromQL allows here — it rejects
|
||||
/// `on`, `ignoring`, `group_left` and `group_right` on a scalar–vector operation. `return_bool` is
|
||||
/// the one modifier that changes what this operation computes: `x > 0` filters the vector and
|
||||
/// `x > bool 0` reports zero or one for each of its values.
|
||||
fn scalar_matching(op: BinaryOp, return_bool: bool) -> BinaryMatching {
|
||||
BinaryMatching {
|
||||
op,
|
||||
return_bool,
|
||||
labels: MatchingLabels::Ignoring(Vec::new()),
|
||||
cardinality: Cardinality::OneToOne,
|
||||
// A fill value fills in a match group that could miss, which a broadcast over a constant never
|
||||
// does: the constant exists at every grid point, so there is no missing side to fill.
|
||||
fill: FillValues::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The binary operator a PromQL token denotes, or `None` when there is no node for it.
|
||||
///
|
||||
/// A comparison is bound like an arithmetic operator, because the node carries `return_bool`
|
||||
/// separately: `x > 0` and `x > bool 0` are one operator with one field changed. [`arithmetic_op`]
|
||||
/// is not reused because it decides what a *fold* may do at compile time and a comparison folds
|
||||
/// nothing, while this map decides what a *node* computes once the query runs.
|
||||
fn binary_op(op: &TokenType) -> Option<BinaryOp> {
|
||||
match op.id() {
|
||||
token::T_ADD => Some(BinaryOp::Add),
|
||||
token::T_SUB => Some(BinaryOp::Sub),
|
||||
token::T_MUL => Some(BinaryOp::Mul),
|
||||
token::T_DIV => Some(BinaryOp::Div),
|
||||
token::T_MOD => Some(BinaryOp::Mod),
|
||||
token::T_POW => Some(BinaryOp::Pow),
|
||||
token::T_ATAN2 => Some(BinaryOp::Atan2),
|
||||
token::T_EQLC => Some(BinaryOp::Eq),
|
||||
token::T_NEQ => Some(BinaryOp::Ne),
|
||||
token::T_GTR => Some(BinaryOp::Gt),
|
||||
token::T_LSS => Some(BinaryOp::Lt),
|
||||
token::T_GTE => Some(BinaryOp::Ge),
|
||||
token::T_LTE => Some(BinaryOp::Le),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Compiles a call: a range function opens a window over the selector it is given, and a math
|
||||
/// function maps the value of its first argument.
|
||||
///
|
||||
@@ -168,9 +377,8 @@ fn compile_call(context: &Context, call: &Call) -> Result<NativeNode> {
|
||||
return window_call(context, func.name, signature, &args.args);
|
||||
}
|
||||
// The name is looked up before the call's arguments are read, so a name this compiler has no
|
||||
// kernel for is refused as such whichever arguments it was given: `pi()` and `time()` have no
|
||||
// vector to map, and the rejection names the name rather than describing a vector they never
|
||||
// wrote.
|
||||
// kernel for is refused as such whichever argument it was given: `absent(x)` is refused as a
|
||||
// name with no kernel, not as a selector this compiler could not compile.
|
||||
let Some((op, vector)) = math_op(func.name, &args.args)? else {
|
||||
return Err(not_implemented(&format!(
|
||||
"a call to `{}`, which has no native kernel",
|
||||
|
||||
@@ -23,7 +23,10 @@ use promql::native::plan::{
|
||||
BoundInstant, BoundMap, BoundWindow, NativeExpr, NativeNode, NativeTree, NativeType,
|
||||
};
|
||||
use promql::range_eval::RangeFunction;
|
||||
use promql_parser::parser::{EvalStmt, Expr as PromExpr, NumberLiteral};
|
||||
use promql::vector_match::{BinaryMatching, BinaryOp};
|
||||
use promql_parser::parser::{
|
||||
BinModifier, EvalStmt, Expr as PromExpr, NumberLiteral, VectorMatchFillValues,
|
||||
};
|
||||
|
||||
use super::compile;
|
||||
|
||||
@@ -70,6 +73,19 @@ fn with_selector_range(range: Duration) -> EvalStmt {
|
||||
stmt
|
||||
}
|
||||
|
||||
/// `query`'s statement with its binary operation's modifier edited by `edit`.
|
||||
///
|
||||
/// The parser rewrites and rejects some modifiers, so a modifier the compiler has to refuse as
|
||||
/// written can only reach it on a tree the parser accepted: this is the way in.
|
||||
fn with_binary_modifier(query: &str, edit: impl FnOnce(&mut BinModifier)) -> EvalStmt {
|
||||
let mut stmt = eval_stmt(query);
|
||||
let PromExpr::Binary(binary) = &mut stmt.expr else {
|
||||
unreachable!("`{query}` parses to a binary operation");
|
||||
};
|
||||
edit(binary.modifier.get_or_insert_with(BinModifier::default));
|
||||
stmt
|
||||
}
|
||||
|
||||
/// `query`'s statement with its call's arguments edited by `edit`.
|
||||
///
|
||||
/// The parser checks arity, so a call with the wrong number of arguments cannot be parsed: the only
|
||||
@@ -140,6 +156,42 @@ fn evaluated(node: &NativeNode) -> &Arc<SeriesSchema> {
|
||||
schema
|
||||
}
|
||||
|
||||
/// The two operands and the matching of a `Binary` node.
|
||||
fn binary_parts(node: &NativeNode) -> (&NativeNode, &NativeNode, &BinaryMatching) {
|
||||
let NativeExpr::Binary {
|
||||
left,
|
||||
right,
|
||||
matching,
|
||||
} = &node.expr
|
||||
else {
|
||||
panic!("the node is not a binary operation: {:?}", node.expr);
|
||||
};
|
||||
(left, right, matching)
|
||||
}
|
||||
|
||||
/// The constant a `Scalar` node binds, if it is one.
|
||||
fn constant(node: &NativeNode) -> Option<f64> {
|
||||
match &node.expr {
|
||||
NativeExpr::Scalar(scalar) => Some(scalar.value),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// The grid spec a `Scalar` node reports, as `(start_ms, step_ms, len)`.
|
||||
fn scalar_grid(node: &NativeNode) -> (i64, i64, u64) {
|
||||
let NativeType::Scalar(grid) = &node.output else {
|
||||
panic!("the node is not a scalar: {:?}", node.output);
|
||||
};
|
||||
let spec = grid.spec();
|
||||
(spec.start_ms, spec.step_ms, spec.len)
|
||||
}
|
||||
|
||||
/// The grid spec a node's evaluated output carries, as `(start_ms, step_ms, len)`.
|
||||
fn grid_of(node: &NativeNode) -> (i64, i64, u64) {
|
||||
let spec = evaluated(node).eval_grid.as_ref().unwrap().spec();
|
||||
(spec.start_ms, spec.step_ms, spec.len)
|
||||
}
|
||||
|
||||
/// `rate(x[5m])` becomes a window over the raw input, on the statement's grid.
|
||||
#[test]
|
||||
fn a_rate_over_a_selector_becomes_a_window_over_raw_input() {
|
||||
@@ -706,13 +758,14 @@ fn the_numeric_edges_of_a_statement_are_bound_as_given() {
|
||||
Expected::Ordering(SeriesOrdering::Contiguous),
|
||||
),
|
||||
// A shape outside the subset is refused for its shape, whatever its timing: the statement
|
||||
// range of an unsupported query is not read, so it cannot turn a refusal into a plan error.
|
||||
// range of an unsupported shape is not read, so it cannot turn a refusal into a plan
|
||||
// error. A subquery is the reachable case of a shape with no node at all.
|
||||
(
|
||||
"an unsupported shape with an inverted range",
|
||||
EvalStmt {
|
||||
end: UNIX_EPOCH,
|
||||
start: UNIX_EPOCH + Duration::from_secs(100),
|
||||
..eval_stmt("x + 1")
|
||||
..eval_stmt("x[5m:]")
|
||||
},
|
||||
SeriesOrdering::KeyThenTime,
|
||||
Expected::Rejected(Rejection::NotImplemented),
|
||||
@@ -810,8 +863,6 @@ fn an_interleaved_scan_is_refused_whatever_the_shape() {
|
||||
#[test]
|
||||
fn an_unsupported_shape_is_rejected_and_names_what_it_found() {
|
||||
let cases = [
|
||||
("x + 1", "the expression `x + 1`"),
|
||||
("sum(x)", "the expression `sum(x)`"),
|
||||
("x[5m:]", "the expression `x[5m:]`"),
|
||||
// Nested shapes are refused by the argument check, not skipped over. A subquery is the
|
||||
// reachable case of an argument that is not a range selector: the parser type-checks
|
||||
@@ -829,10 +880,6 @@ fn an_unsupported_shape_is_rejected_and_names_what_it_found() {
|
||||
"absent(x)",
|
||||
"a call to `absent`, which has no native kernel",
|
||||
),
|
||||
// The two functions PromQL calls with no arguments at all: neither has a kernel, and the
|
||||
// rejection names the name rather than describing a vector they were never given.
|
||||
("pi()", "a call to `pi`, which has no native kernel"),
|
||||
("time()", "a call to `time`, which has no native kernel"),
|
||||
];
|
||||
|
||||
for (query, named) in cases {
|
||||
@@ -876,6 +923,45 @@ fn the_display_of_a_compiled_tree_is_the_tree() {
|
||||
);
|
||||
}
|
||||
|
||||
/// The display of a tree the batch added nodes to, checked as a whole: the shapes are read by
|
||||
/// someone reading an `EXPLAIN`, so a wrong nesting, operator, constant or grouping has to show up on
|
||||
/// a line rather than only in an assertion over the tree.
|
||||
#[test]
|
||||
fn the_display_of_a_tree_the_batch_added_to_is_the_query() {
|
||||
let cases: [(&str, &[&str]); 2] = [
|
||||
(
|
||||
"-x",
|
||||
&[
|
||||
"NativeTree",
|
||||
" root Binary * ignoring [] one-to-one",
|
||||
" left Scalar -1",
|
||||
" right Instant lookback 5000ms offset 0 on grid 0/5000x21",
|
||||
" input Input raw samples",
|
||||
],
|
||||
),
|
||||
(
|
||||
"0 < x",
|
||||
&[
|
||||
"NativeTree",
|
||||
" root Binary < ignoring [] one-to-one",
|
||||
" left Scalar 0",
|
||||
" right Instant lookback 5000ms offset 0 on grid 0/5000x21",
|
||||
" input Input raw samples",
|
||||
],
|
||||
),
|
||||
];
|
||||
|
||||
for (query, expected) in cases {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
assert_eq!(
|
||||
tree.to_string(),
|
||||
expected.join("\n"),
|
||||
"`{query}` is displayed as a different query"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Parentheses group; they do not change what a statement asks for. A compiler that refused them
|
||||
/// would reject a query on how it was typed rather than on what it means, and a statement written
|
||||
/// with them is otherwise exactly the shape the compiler accepts.
|
||||
@@ -1159,3 +1245,405 @@ fn a_math_function_over_a_non_vector_argument_is_refused_and_names_it() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A constant on the right of a vector binds as the number it is, next to the vector the rest of the
|
||||
/// query compiles to: `x * 100` is `x`'s instant selection multiplied by `100`.
|
||||
#[test]
|
||||
fn a_constant_binds_as_a_scalar_next_to_the_vector() {
|
||||
let input = raw_schema();
|
||||
let tree = compile(
|
||||
&eval_stmt("x * 100"),
|
||||
Arc::clone(&input),
|
||||
SeriesOrdering::KeyThenTime,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let (left, right, matching) = binary_parts(tree.root());
|
||||
assert_eq!(matching.op, BinaryOp::Mul);
|
||||
assert!(
|
||||
!matching.return_bool,
|
||||
"`x * 100` is not a comparison and has no bool"
|
||||
);
|
||||
|
||||
let NativeExpr::Instant { .. } = &left.expr else {
|
||||
panic!(
|
||||
"the left operand is not the selector's instant: {:?}",
|
||||
left.expr
|
||||
);
|
||||
};
|
||||
assert_eq!(
|
||||
constant(right),
|
||||
Some(100.0),
|
||||
"the right operand is not the constant the query wrote"
|
||||
);
|
||||
assert_eq!(
|
||||
scalar_grid(right),
|
||||
(0, 5_000, 21),
|
||||
"the constant is not on the statement's grid"
|
||||
);
|
||||
|
||||
// A scalar carries no fields, so the operation reports the vector side's layout and grid.
|
||||
let output = evaluated(tree.root());
|
||||
assert!(
|
||||
Arc::ptr_eq(&output.fields, &input.fields),
|
||||
"the broadcast did not keep the vector's fields"
|
||||
);
|
||||
assert_eq!(
|
||||
grid_of(tree.root()),
|
||||
(0, 5_000, 21),
|
||||
"the operation does not land on the statement's grid"
|
||||
);
|
||||
}
|
||||
|
||||
/// The operands keep the order they were written in, because most operators are not commutative:
|
||||
/// `1 / x` is not `x / 1`. The constant side is what tells them apart, so a compiler that sorted the
|
||||
/// operands would compile both queries to the same, wrong tree.
|
||||
#[test]
|
||||
fn a_non_commutative_operation_keeps_the_operands_where_they_were_written() {
|
||||
let cases = [
|
||||
// (query, operator, the constant, whether the constant is on the left)
|
||||
("1 / x", BinaryOp::Div, 1.0, true),
|
||||
("x / 1", BinaryOp::Div, 1.0, false),
|
||||
("2 - x", BinaryOp::Sub, 2.0, true),
|
||||
("x - 2", BinaryOp::Sub, 2.0, false),
|
||||
("x % 2", BinaryOp::Mod, 2.0, false),
|
||||
("2 ^ x", BinaryOp::Pow, 2.0, true),
|
||||
];
|
||||
|
||||
for (query, op, value, constant_on_the_left) in cases {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
let (left, right, matching) = binary_parts(tree.root());
|
||||
assert_eq!(matching.op, op, "`{query}` bound the wrong operator");
|
||||
let (scalar_side, vector_side) = if constant_on_the_left {
|
||||
(left, right)
|
||||
} else {
|
||||
(right, left)
|
||||
};
|
||||
assert_eq!(
|
||||
constant(scalar_side),
|
||||
Some(value),
|
||||
"`{query}` put the constant on the wrong side"
|
||||
);
|
||||
assert!(
|
||||
!matches!(vector_side.expr, NativeExpr::Scalar(_)),
|
||||
"`{query}` compiled the vector operand to a constant: {:?}",
|
||||
vector_side.expr
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A binary operation compiles by compiling both operands, so it nests as deeply as the query was
|
||||
/// written: `2 / (x + 1)` is a division whose right operand is the addition, not a shape the
|
||||
/// compiler knows by name.
|
||||
#[test]
|
||||
fn a_binary_operation_nests_over_the_operands_it_was_written_with() {
|
||||
let tree = compile(
|
||||
&eval_stmt("2 / (x + 1)"),
|
||||
raw_schema(),
|
||||
SeriesOrdering::KeyThenTime,
|
||||
)
|
||||
.unwrap_or_else(|error| panic!("`2 / (x + 1)` was refused: {error}"));
|
||||
|
||||
let (left, right, matching) = binary_parts(tree.root());
|
||||
assert_eq!(matching.op, BinaryOp::Div);
|
||||
assert_eq!(constant(left), Some(2.0), "the divisor was not bound");
|
||||
|
||||
let (inner_left, inner_right, inner_matching) = binary_parts(right);
|
||||
assert_eq!(inner_matching.op, BinaryOp::Add);
|
||||
let NativeExpr::Instant { .. } = &inner_left.expr else {
|
||||
panic!(
|
||||
"the sum's left operand is not the selector: {:?}",
|
||||
inner_left.expr
|
||||
);
|
||||
};
|
||||
assert_eq!(
|
||||
constant(inner_right),
|
||||
Some(1.0),
|
||||
"the summand was not bound as a constant"
|
||||
);
|
||||
|
||||
// Every node is on the statement's grid, which is what lets the operation align them.
|
||||
assert_eq!(grid_of(tree.root()), (0, 5_000, 21));
|
||||
assert_eq!(grid_of(right), (0, 5_000, 21));
|
||||
}
|
||||
|
||||
/// `return_bool` is a field of the operation rather than a different tree: `x > 0` filters and
|
||||
/// `x > bool 0` reports zero or one, and the two differ in nothing else. Both directions of a
|
||||
/// comparison are asserted, because which side the constant is on decides what the filter keeps and
|
||||
/// a compiler that always put the constant on one side would compile `0 < x` as `x > 0`.
|
||||
#[test]
|
||||
fn a_comparison_carries_whether_it_filters_or_reports_bool() {
|
||||
let cases = [
|
||||
("x > 0", BinaryOp::Gt, false),
|
||||
("x > bool 0", BinaryOp::Gt, true),
|
||||
("0 < x", BinaryOp::Lt, false),
|
||||
("0 < bool x", BinaryOp::Lt, true),
|
||||
("x == 0", BinaryOp::Eq, false),
|
||||
("x != 0", BinaryOp::Ne, false),
|
||||
("x >= 0", BinaryOp::Ge, false),
|
||||
("x <= 0", BinaryOp::Le, false),
|
||||
];
|
||||
|
||||
for (query, op, return_bool) in cases {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
let (_, _, matching) = binary_parts(tree.root());
|
||||
assert_eq!(matching.op, op, "`{query}` bound the wrong operator");
|
||||
assert_eq!(
|
||||
matching.return_bool, return_bool,
|
||||
"`{query}` bound the wrong comparison mode"
|
||||
);
|
||||
}
|
||||
|
||||
// The two modes are one tree otherwise: the query text is what differs, and a tree that recorded
|
||||
// the mode somewhere else would fail the assertions above rather than this one.
|
||||
let filters = compile(
|
||||
&eval_stmt("x > 0"),
|
||||
raw_schema(),
|
||||
SeriesOrdering::KeyThenTime,
|
||||
)
|
||||
.unwrap();
|
||||
let reports = compile(
|
||||
&eval_stmt("x > bool 0"),
|
||||
raw_schema(),
|
||||
SeriesOrdering::KeyThenTime,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(grid_of(filters.root()), grid_of(reports.root()));
|
||||
}
|
||||
|
||||
/// A negation binds as the multiplication it is: `-x` is `-1 * x`. The tree has no negate node, and
|
||||
/// the multiplication kernel computes exactly the value the query asks for.
|
||||
#[test]
|
||||
fn a_negation_binds_as_a_multiplication_by_minus_one() {
|
||||
for query in ["-x", "-(x)"] {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
let (left, right, matching) = binary_parts(tree.root());
|
||||
assert_eq!(
|
||||
matching.op,
|
||||
BinaryOp::Mul,
|
||||
"`{query}` is not bound as a multiplication"
|
||||
);
|
||||
assert_eq!(
|
||||
constant(left),
|
||||
Some(-1.0),
|
||||
"`{query}` did not bind `-1` on the left"
|
||||
);
|
||||
let NativeExpr::Instant { .. } = &right.expr else {
|
||||
panic!("`{query}`'s operand is not the selector: {:?}", right.expr);
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// The negation applies to whatever its operand compiles to, so a window can be negated: `-rate(x[5m])`
|
||||
/// is a multiplication over the window rather than a new shape.
|
||||
#[test]
|
||||
fn a_negation_applies_to_the_node_its_operand_compiles_to() {
|
||||
let tree = compile(
|
||||
&eval_stmt("-rate(x[5m])"),
|
||||
raw_schema(),
|
||||
SeriesOrdering::KeyThenTime,
|
||||
)
|
||||
.unwrap_or_else(|error| panic!("`-rate(x[5m])` was refused: {error}"));
|
||||
|
||||
let (left, right, matching) = binary_parts(tree.root());
|
||||
assert_eq!(matching.op, BinaryOp::Mul);
|
||||
assert_eq!(constant(left), Some(-1.0));
|
||||
let NativeExpr::Window { op, .. } = &right.expr else {
|
||||
panic!("the negation's operand is not a window: {:?}", right.expr);
|
||||
};
|
||||
assert_eq!(op.function, RangeFunction::Rate);
|
||||
assert_eq!(op.window.range_ms, 300_000);
|
||||
}
|
||||
|
||||
/// A constant written as an expression binds the number it folds to, by the fold a call's parameter
|
||||
/// already uses. The value is asserted exactly, so a fold that produced a neighbouring number would
|
||||
/// fail here.
|
||||
#[test]
|
||||
fn a_constant_operand_binds_the_value_it_folds_to() {
|
||||
let cases = [
|
||||
("x * (1 + 2)", 3.0),
|
||||
("x + 2 ^ 10", 1024.0),
|
||||
// `^` binds tighter than `*`, so this is `x * 8` rather than `(x * 2) ^ 3`.
|
||||
("x * 2 ^ 3", 8.0),
|
||||
("x * -0.5", -0.5),
|
||||
("x * (1 - 0.1)", 0.9),
|
||||
];
|
||||
|
||||
for (query, value) in cases {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
let (left, right, _) = binary_parts(tree.root());
|
||||
let scalar_side = if constant(left).is_some() {
|
||||
left
|
||||
} else {
|
||||
right
|
||||
};
|
||||
assert_eq!(
|
||||
constant(scalar_side),
|
||||
Some(value),
|
||||
"`{query}` did not bind the number it folds to"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// An operation between two vectors is refused because the tree cannot say which selector each
|
||||
/// `Input` reads: `a + b` and `a + a` would compile to the same tree. The refusal names the missing
|
||||
/// identity rather than the query it found, so whoever adds it knows what to add.
|
||||
#[test]
|
||||
fn an_operation_between_two_vectors_is_refused_for_the_missing_selector_identity() {
|
||||
for query in [
|
||||
"a + b",
|
||||
"a > b",
|
||||
"rate(a[5m]) * rate(b[5m])",
|
||||
"abs(a) + abs(b)",
|
||||
] {
|
||||
let error = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.expect_err(&format!("`{query}` compiled, and it should not have"));
|
||||
assert!(
|
||||
matches!(error, DataFusionError::NotImplemented(_)),
|
||||
"`{query}` did not report NotImplemented: {error}"
|
||||
);
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains("between two vectors"),
|
||||
"`{query}` did not say what it found: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains("selector"),
|
||||
"`{query}` did not name what an input is missing: {message}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A set operator has no node at all, so it is refused as the set operator it is rather than as an
|
||||
/// arithmetic operation the compiler could not bind: `and`, `or` and `unless` combine two vectors'
|
||||
/// series, which is not what a binary value node does.
|
||||
#[test]
|
||||
fn a_set_operator_is_refused_as_a_set_operator() {
|
||||
for (query, name) in [
|
||||
("a and b", "and"),
|
||||
("a or b", "or"),
|
||||
("a unless b", "unless"),
|
||||
] {
|
||||
let error = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.expect_err(&format!("`{query}` compiled, and it should not have"));
|
||||
assert!(
|
||||
matches!(error, DataFusionError::NotImplemented(_)),
|
||||
"`{query}` did not report NotImplemented: {error}"
|
||||
);
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains(&format!("set operator `{name}`")),
|
||||
"`{query}` did not name the operator it found: {message}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A statement with no vector has nothing to read, so it is refused: a tree reads a scan and reports
|
||||
/// evaluated samples, and a constant is bound next to a vector rather than on its own. The refusal
|
||||
/// names the expression and its type, because the type is why it cannot be bound.
|
||||
#[test]
|
||||
fn a_statement_with_no_vector_is_refused() {
|
||||
// `pi()` and `time()` are the two functions PromQL calls with no arguments. Both are scalars, so
|
||||
// they reach this refusal before any function-name lookup: what a tree cannot do with them is
|
||||
// report them, not fold them. The rejection still names the expression it found, so a reader
|
||||
// learns which function it was.
|
||||
for (query, found) in [
|
||||
("1 + 2", "which is a scalar"),
|
||||
("1", "which is a scalar"),
|
||||
("-3", "which is a scalar"),
|
||||
("pi()", "which is a scalar"),
|
||||
("time()", "which is a scalar"),
|
||||
] {
|
||||
let error = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.expect_err(&format!("`{query}` compiled, and it should not have"));
|
||||
assert!(
|
||||
matches!(error, DataFusionError::NotImplemented(_)),
|
||||
"`{query}` did not report NotImplemented: {error}"
|
||||
);
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains(&format!("`{query}`")),
|
||||
"`{query}` did not name the expression it found: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains(found),
|
||||
"`{query}` did not name the type it found: {message}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A vector-matching modifier has nothing to match when one operand is a constant, and the tree's
|
||||
/// matching has no field for a fill value. Both are refused rather than dropped: dropping one would
|
||||
/// compile a query into an operation other than the one it wrote. `bool` is not a matching modifier
|
||||
/// and is bound, which is why it is not in this list.
|
||||
#[test]
|
||||
fn a_vector_matching_modifier_over_a_constant_is_refused() {
|
||||
let cases = [
|
||||
(eval_stmt("x + ignoring() 1"), "x + ignoring() 1"),
|
||||
(eval_stmt("x + on() group_left 1"), "x + on() group_left 1"),
|
||||
(
|
||||
eval_stmt("x + on() group_right 1"),
|
||||
"x + on() group_right 1",
|
||||
),
|
||||
// A fill value is a matching extension the tree has no field for, so a fill is refused
|
||||
// rather than bound without it. The parser accepts a fill only between two vectors, so the
|
||||
// value is set on a tree the parser accepted.
|
||||
(
|
||||
with_binary_modifier("x + 1", |modifier| {
|
||||
modifier.fill_values = VectorMatchFillValues::new(0.0, 0.0);
|
||||
}),
|
||||
"x + 1 with a fill of 0 on both sides",
|
||||
),
|
||||
];
|
||||
|
||||
for (stmt, query) in cases {
|
||||
let error = compile(&stmt, raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.expect_err(&format!("`{query}` compiled, and it should not have"));
|
||||
assert!(
|
||||
matches!(error, DataFusionError::NotImplemented(_)),
|
||||
"`{query}` did not report NotImplemented: {error}"
|
||||
);
|
||||
let message = error.to_string();
|
||||
assert!(
|
||||
message.contains("vector matching"),
|
||||
"`{query}` did not say what it found: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains("constant operand"),
|
||||
"`{query}` blamed the wrong operand: {message}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The shared acceptance queries of this batch, compiled whole: each one combines the shapes the
|
||||
/// batch added, so a compiler that bound any of them wrongly would fail here rather than in a
|
||||
/// narrower test.
|
||||
#[test]
|
||||
fn the_batch_acceptance_queries_compile() {
|
||||
let cases = [
|
||||
// Recursion plus a constant on the left.
|
||||
"2 / (x + 1)",
|
||||
// Filter against bool comparison, in both directions.
|
||||
"x > 0",
|
||||
"0 < x",
|
||||
"x > bool 0",
|
||||
// A negation.
|
||||
"-x",
|
||||
];
|
||||
|
||||
for query in cases {
|
||||
let tree = compile(&eval_stmt(query), raw_schema(), SeriesOrdering::KeyThenTime)
|
||||
.unwrap_or_else(|error| panic!("`{query}` was refused: {error}"));
|
||||
// Every tree the compiler accepts reports a grid, whether it ends in a comparison or a
|
||||
// scalar broadcast.
|
||||
assert_eq!(
|
||||
grid_of(tree.root()),
|
||||
(0, 5_000, 21),
|
||||
"`{query}` does not land on the statement's grid"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user