mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-05-21 07:20:41 +00:00
feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: upgrading network deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adapt new df's trait requirements Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi <jiachun_feng@proton.me> * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me>
This commit is contained in:
@@ -14,6 +14,7 @@
|
||||
|
||||
pub use sqlparser::ast::{
|
||||
visit_expressions_mut, visit_statements_mut, BinaryOperator, ColumnDef, ColumnOption,
|
||||
ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName,
|
||||
SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor, VisitorMut,
|
||||
ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArguments,
|
||||
Ident, ObjectName, SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor,
|
||||
VisitorMut,
|
||||
};
|
||||
|
||||
@@ -72,7 +72,7 @@ pub enum Error {
|
||||
},
|
||||
|
||||
// Syntax error from sql parser.
|
||||
#[snafu(display(""))]
|
||||
#[snafu(display("Invalid SQL syntax"))]
|
||||
Syntax {
|
||||
#[snafu(source)]
|
||||
error: ParserError,
|
||||
@@ -81,7 +81,7 @@ pub enum Error {
|
||||
},
|
||||
|
||||
// Syntax error from tql parser.
|
||||
#[snafu(display(""))]
|
||||
#[snafu(display("Invalid TQL syntax"))]
|
||||
TQLSyntax {
|
||||
#[snafu(source)]
|
||||
error: TQLError,
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
|
||||
#![feature(box_patterns)]
|
||||
#![feature(assert_matches)]
|
||||
#![feature(if_let_guard)]
|
||||
#![feature(let_chains)]
|
||||
|
||||
pub mod ast;
|
||||
|
||||
@@ -50,7 +50,7 @@ impl ParserContext<'_> {
|
||||
|
||||
/// Parses parser context to Query.
|
||||
pub fn parser_query(&mut self) -> Result<Box<Query>> {
|
||||
Ok(Box::new(self.parser.parse_query().context(SyntaxSnafu)?))
|
||||
self.parser.parse_query().context(SyntaxSnafu)
|
||||
}
|
||||
|
||||
/// Parses SQL with given dialect
|
||||
|
||||
@@ -46,6 +46,8 @@ impl ParserContext<'_> {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use sqlparser::ast::FunctionArguments;
|
||||
|
||||
use super::*;
|
||||
use crate::ast::{Expr, Function, FunctionArg, FunctionArgExpr, Value};
|
||||
use crate::dialect::GreptimeDbDialect;
|
||||
@@ -62,9 +64,12 @@ mod tests {
|
||||
let stmt = result.remove(0);
|
||||
match &stmt {
|
||||
Statement::Admin(Admin::Func(Function { name, args, .. })) => {
|
||||
let FunctionArguments::List(arg_list) = args else {
|
||||
unreachable!()
|
||||
};
|
||||
assert_eq!("flush_table", name.to_string());
|
||||
assert_eq!(args.len(), 1);
|
||||
assert!(matches!(&args[0],
|
||||
assert_eq!(arg_list.args.len(), 1);
|
||||
assert!(matches!(&arg_list.args[0],
|
||||
FunctionArg::Unnamed(FunctionArgExpr::Expr(
|
||||
Expr::Value(Value::SingleQuotedString(s))
|
||||
)) if s == "test"));
|
||||
@@ -86,8 +91,11 @@ mod tests {
|
||||
let stmt = result.remove(0);
|
||||
match &stmt {
|
||||
Statement::Admin(Admin::Func(Function { name, args, .. })) => {
|
||||
let FunctionArguments::List(arg_list) = args else {
|
||||
unreachable!()
|
||||
};
|
||||
assert_eq!("test", name.to_string());
|
||||
assert_eq!(args.len(), 0);
|
||||
assert_eq!(arg_list.args.len(), 0);
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
|
||||
@@ -644,7 +644,7 @@ mod tests {
|
||||
let err = result.output_msg();
|
||||
assert_eq!(
|
||||
err,
|
||||
"sql parser error: Expected COLUMN, found: a at Line: 1, Column 30"
|
||||
"Invalid SQL syntax: sql parser error: Expected: COLUMN, found: a at Line: 1, Column: 30"
|
||||
);
|
||||
|
||||
let sql = "ALTER TABLE my_metric_1 DROP COLUMN a";
|
||||
@@ -790,7 +790,7 @@ mod tests {
|
||||
ParserContext::create_with_dialect(sql, &GreptimeDbDialect {}, ParseOptions::default())
|
||||
.unwrap_err();
|
||||
let err = result.output_msg();
|
||||
assert_eq!(err, "sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t");
|
||||
assert_eq!(err, "Invalid SQL syntax: sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t");
|
||||
|
||||
let sql = "ALTER TABLE test_table RENAME table_t";
|
||||
let mut result =
|
||||
|
||||
@@ -44,9 +44,8 @@ use crate::statements::create::{
|
||||
CreateTableLike, CreateView, Partitions, TableConstraint, VECTOR_OPT_DIM,
|
||||
};
|
||||
use crate::statements::statement::Statement;
|
||||
use crate::statements::{
|
||||
get_data_type_by_alias_name, sql_data_type_to_concrete_data_type, OptionMap,
|
||||
};
|
||||
use crate::statements::transform::type_alias::get_data_type_by_alias_name;
|
||||
use crate::statements::{sql_data_type_to_concrete_data_type, OptionMap};
|
||||
use crate::util::parse_option_string;
|
||||
|
||||
pub const ENGINE: &str = "ENGINE";
|
||||
@@ -291,13 +290,7 @@ impl<'a> ParserContext<'a> {
|
||||
reason: format!("cannot cast {} to interval type", expire_after_expr),
|
||||
})?;
|
||||
if let ScalarValue::IntervalMonthDayNano(Some(nanoseconds)) = expire_after_lit {
|
||||
Some(
|
||||
i64::try_from(nanoseconds / 1_000_000_000)
|
||||
.ok()
|
||||
.with_context(|| InvalidIntervalSnafu {
|
||||
reason: format!("interval {} overflows", nanoseconds),
|
||||
})?,
|
||||
)
|
||||
Some(nanoseconds.nanoseconds / 1_000_000_000)
|
||||
} else {
|
||||
unreachable!()
|
||||
}
|
||||
@@ -326,7 +319,7 @@ impl<'a> ParserContext<'a> {
|
||||
.expect_keyword(Keyword::AS)
|
||||
.context(SyntaxSnafu)?;
|
||||
|
||||
let query = Box::new(self.parser.parse_query().context(error::SyntaxSnafu)?);
|
||||
let query = self.parser.parse_query().context(error::SyntaxSnafu)?;
|
||||
|
||||
Ok(Statement::CreateFlow(CreateFlow {
|
||||
flow_name,
|
||||
@@ -1179,7 +1172,7 @@ mod tests {
|
||||
assert_column_def(&columns[0].column_def, "host", "STRING");
|
||||
assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP");
|
||||
assert_column_def(&columns[2].column_def, "cpu", "FLOAT");
|
||||
assert_column_def(&columns[3].column_def, "memory", "FLOAT64");
|
||||
assert_column_def(&columns[3].column_def, "memory", "DOUBLE");
|
||||
|
||||
let constraints = &c.constraints;
|
||||
assert_eq!(
|
||||
@@ -1758,7 +1751,7 @@ ENGINE=mito";
|
||||
assert!(result
|
||||
.unwrap_err()
|
||||
.output_msg()
|
||||
.contains("sql parser error: Expected ON, found: COLUMNS"));
|
||||
.contains("sql parser error: Expected: ON, found: COLUMNS"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1835,7 +1828,7 @@ ENGINE=mito";
|
||||
assert_column_def(&columns[0].column_def, "host", "STRING");
|
||||
assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP");
|
||||
assert_column_def(&columns[2].column_def, "cpu", "FLOAT");
|
||||
assert_column_def(&columns[3].column_def, "memory", "FLOAT64");
|
||||
assert_column_def(&columns[3].column_def, "memory", "DOUBLE");
|
||||
|
||||
let constraints = &c.constraints;
|
||||
assert_eq!(
|
||||
|
||||
@@ -68,12 +68,13 @@ mod tests {
|
||||
with_hints: vec![],
|
||||
partitions: vec![],
|
||||
version: None,
|
||||
with_ordinality: false,
|
||||
},
|
||||
joins: vec![],
|
||||
}],
|
||||
lateral_views: vec![],
|
||||
selection: None,
|
||||
group_by: GroupByExpr::Expressions(vec![]),
|
||||
group_by: GroupByExpr::Expressions(vec![], vec![]),
|
||||
cluster_by: vec![],
|
||||
distribute_by: vec![],
|
||||
sort_by: vec![],
|
||||
@@ -81,18 +82,24 @@ mod tests {
|
||||
qualify: None,
|
||||
named_window: vec![],
|
||||
value_table_mode: None,
|
||||
top_before_distinct: false,
|
||||
prewhere: None,
|
||||
window_before_qualify: false,
|
||||
connect_by: None,
|
||||
};
|
||||
|
||||
let sp_statement = SpStatement::Query(Box::new(SpQuery {
|
||||
with: None,
|
||||
body: Box::new(sqlparser::ast::SetExpr::Select(Box::new(select))),
|
||||
order_by: vec![],
|
||||
order_by: None,
|
||||
limit: None,
|
||||
limit_by: vec![],
|
||||
offset: None,
|
||||
fetch: None,
|
||||
locks: vec![],
|
||||
for_clause: None,
|
||||
settings: None,
|
||||
format_clause: None,
|
||||
}));
|
||||
|
||||
let explain = Explain::try_from(SpStatement::Explain {
|
||||
@@ -101,6 +108,8 @@ mod tests {
|
||||
verbose: false,
|
||||
statement: Box::new(sp_statement),
|
||||
format: None,
|
||||
query_plan: false,
|
||||
options: None,
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ impl ParserContext<'_> {
|
||||
pub(crate) fn parse_query(&mut self) -> Result<Statement> {
|
||||
let spquery = self.parser.parse_query().context(error::SyntaxSnafu)?;
|
||||
|
||||
Ok(Statement::Query(Box::new(Query::try_from(spquery)?)))
|
||||
Ok(Statement::Query(Box::new(Query::try_from(*spquery)?)))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -56,6 +56,6 @@ mod tests {
|
||||
assert!(result
|
||||
.unwrap_err()
|
||||
.output_msg()
|
||||
.contains("Expected an expression"));
|
||||
.contains("Expected: an expression"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,11 +28,14 @@ impl ParserContext<'_> {
|
||||
let spstatement = self.parser.parse_set().context(error::SyntaxSnafu)?;
|
||||
match spstatement {
|
||||
SpStatement::SetVariable {
|
||||
variable,
|
||||
variables,
|
||||
value,
|
||||
hivevar,
|
||||
..
|
||||
} if !hivevar => Ok(Statement::SetVariables(SetVariables { variable, value })),
|
||||
} if !hivevar => Ok(Statement::SetVariables(SetVariables {
|
||||
variable: (*variables)[0].clone(),
|
||||
value,
|
||||
})),
|
||||
|
||||
SpStatement::SetTimeZone { value, .. } => Ok(Statement::SetVariables(SetVariables {
|
||||
variable: ObjectName(vec![Ident {
|
||||
|
||||
@@ -18,6 +18,7 @@ use chrono::Utc;
|
||||
use datafusion::config::ConfigOptions;
|
||||
use datafusion::error::Result as DfResult;
|
||||
use datafusion::execution::context::SessionState;
|
||||
use datafusion::execution::SessionStateBuilder;
|
||||
use datafusion::optimizer::simplify_expressions::ExprSimplifier;
|
||||
use datafusion_common::{DFSchema, ScalarValue};
|
||||
use datafusion_expr::execution_props::ExecutionProps;
|
||||
@@ -72,7 +73,11 @@ struct StubContextProvider {
|
||||
impl Default for StubContextProvider {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
state: SessionState::new_with_config_rt(Default::default(), Default::default()),
|
||||
state: SessionStateBuilder::new()
|
||||
.with_config(Default::default())
|
||||
.with_runtime_env(Default::default())
|
||||
.with_default_features()
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ pub mod set_variables;
|
||||
pub mod show;
|
||||
pub mod statement;
|
||||
pub mod tql;
|
||||
mod transform;
|
||||
pub(crate) mod transform;
|
||||
pub mod truncate;
|
||||
|
||||
use std::str::FromStr;
|
||||
@@ -61,7 +61,7 @@ use crate::error::{
|
||||
};
|
||||
use crate::statements::create::Column;
|
||||
pub use crate::statements::option_map::OptionMap;
|
||||
pub use crate::statements::transform::{get_data_type_by_alias_name, transform_statements};
|
||||
pub(crate) use crate::statements::transform::transform_statements;
|
||||
|
||||
const VECTOR_TYPE_NAME: &str = "VECTOR";
|
||||
|
||||
|
||||
@@ -695,8 +695,8 @@ CREATE TABLE t2 LIKE t1"#,
|
||||
CREATE EXTERNAL TABLE city (
|
||||
host STRING,
|
||||
ts TIMESTAMP,
|
||||
cpu FLOAT64 DEFAULT 0,
|
||||
memory FLOAT64,
|
||||
cpu DOUBLE DEFAULT 0,
|
||||
memory DOUBLE,
|
||||
TIME INDEX (ts),
|
||||
PRIMARY KEY (host)
|
||||
)
|
||||
|
||||
@@ -13,7 +13,9 @@
|
||||
// limitations under the License.
|
||||
|
||||
use serde::Serialize;
|
||||
use sqlparser::ast::{ObjectName, Query, SetExpr, Statement, UnaryOperator, Values};
|
||||
use sqlparser::ast::{
|
||||
Insert as SpInsert, ObjectName, Query, SetExpr, Statement, UnaryOperator, Values,
|
||||
};
|
||||
use sqlparser::parser::ParserError;
|
||||
use sqlparser_derive::{Visit, VisitMut};
|
||||
|
||||
@@ -39,14 +41,14 @@ macro_rules! parse_fail {
|
||||
impl Insert {
|
||||
pub fn table_name(&self) -> &ObjectName {
|
||||
match &self.inner {
|
||||
Statement::Insert { table_name, .. } => table_name,
|
||||
Statement::Insert(insert) => &insert.table_name,
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn columns(&self) -> Vec<&String> {
|
||||
match &self.inner {
|
||||
Statement::Insert { columns, .. } => columns.iter().map(|ident| &ident.value).collect(),
|
||||
Statement::Insert(insert) => insert.columns.iter().map(|ident| &ident.value).collect(),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
@@ -54,14 +56,14 @@ impl Insert {
|
||||
/// Extracts the literal insert statement body if possible
|
||||
pub fn values_body(&self) -> Result<Vec<Vec<Value>>> {
|
||||
match &self.inner {
|
||||
Statement::Insert {
|
||||
Statement::Insert(SpInsert {
|
||||
source:
|
||||
Some(box Query {
|
||||
body: box SetExpr::Values(Values { rows, .. }),
|
||||
..
|
||||
}),
|
||||
..
|
||||
} => sql_exprs_to_values(rows),
|
||||
}) => sql_exprs_to_values(rows),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
}
|
||||
@@ -70,14 +72,14 @@ impl Insert {
|
||||
/// The rules is the same as function `values_body()`.
|
||||
pub fn can_extract_values(&self) -> bool {
|
||||
match &self.inner {
|
||||
Statement::Insert {
|
||||
Statement::Insert(SpInsert {
|
||||
source:
|
||||
Some(box Query {
|
||||
body: box SetExpr::Values(Values { rows, .. }),
|
||||
..
|
||||
}),
|
||||
..
|
||||
} => rows.iter().all(|es| {
|
||||
}) => rows.iter().all(|es| {
|
||||
es.iter().all(|expr| match expr {
|
||||
Expr::Value(_) => true,
|
||||
Expr::Identifier(ident) => {
|
||||
@@ -100,10 +102,10 @@ impl Insert {
|
||||
|
||||
pub fn query_body(&self) -> Result<Option<GtQuery>> {
|
||||
Ok(match &self.inner {
|
||||
Statement::Insert {
|
||||
Statement::Insert(SpInsert {
|
||||
source: Some(box query),
|
||||
..
|
||||
} => Some(query.clone().try_into()?),
|
||||
}) => Some(query.clone().try_into()?),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -12,20 +12,19 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
mod expand_interval;
|
||||
pub(crate) mod type_alias;
|
||||
|
||||
use std::ops::ControlFlow;
|
||||
use std::sync::Arc;
|
||||
|
||||
use expand_interval::ExpandIntervalTransformRule;
|
||||
use lazy_static::lazy_static;
|
||||
use sqlparser::ast::{visit_expressions_mut, Expr};
|
||||
use type_alias::TypeAliasTransformRule;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::statements::statement::Statement;
|
||||
mod expand_interval;
|
||||
mod type_alias;
|
||||
|
||||
use expand_interval::ExpandIntervalTransformRule;
|
||||
pub use type_alias::get_data_type_by_alias_name;
|
||||
use type_alias::TypeAliasTransformRule;
|
||||
|
||||
lazy_static! {
|
||||
/// [TransformRule] registry
|
||||
|
||||
@@ -371,6 +371,7 @@ mod tests {
|
||||
expr: single_quoted_string_expr("3y2mon".to_string()),
|
||||
data_type: DataType::Interval,
|
||||
format: None,
|
||||
kind: sqlparser::ast::CastKind::Cast,
|
||||
};
|
||||
|
||||
let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_interval_expr);
|
||||
@@ -393,6 +394,7 @@ mod tests {
|
||||
expr: single_quoted_string_expr("5".to_string()),
|
||||
data_type: DataType::Int64,
|
||||
format: None,
|
||||
kind: sqlparser::ast::CastKind::Cast,
|
||||
};
|
||||
let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_i64_expr);
|
||||
assert_eq!(control_flow, ControlFlow::Continue(()));
|
||||
@@ -402,6 +404,7 @@ mod tests {
|
||||
expr: single_quoted_string_expr("5".to_string()),
|
||||
data_type: DataType::Int64,
|
||||
format: None,
|
||||
kind: sqlparser::ast::CastKind::Cast,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
@@ -16,7 +16,8 @@ use std::ops::ControlFlow;
|
||||
|
||||
use datatypes::data_type::DataType as GreptimeDataType;
|
||||
use sqlparser::ast::{
|
||||
DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName, Value,
|
||||
DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArgumentList, Ident,
|
||||
ObjectName, Value,
|
||||
};
|
||||
|
||||
use crate::error::Result;
|
||||
@@ -75,18 +76,21 @@ impl TransformRule for TypeAliasTransformRule {
|
||||
fn cast_expr_to_arrow_cast_func(expr: Expr, cast_type: String) -> Function {
|
||||
Function {
|
||||
name: ObjectName(vec![Ident::new("arrow_cast")]),
|
||||
args: vec![
|
||||
FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)),
|
||||
FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
|
||||
Value::SingleQuotedString(cast_type),
|
||||
))),
|
||||
],
|
||||
args: sqlparser::ast::FunctionArguments::List(FunctionArgumentList {
|
||||
args: vec![
|
||||
FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)),
|
||||
FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
|
||||
Value::SingleQuotedString(cast_type),
|
||||
))),
|
||||
],
|
||||
duplicate_treatment: None,
|
||||
clauses: vec![],
|
||||
}),
|
||||
filter: None,
|
||||
null_treatment: None,
|
||||
over: None,
|
||||
distinct: false,
|
||||
special: false,
|
||||
order_by: vec![],
|
||||
parameters: sqlparser::ast::FunctionArguments::None,
|
||||
within_group: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
@@ -99,30 +103,14 @@ impl TransformRule for TypeAliasTransformRule {
|
||||
expr: cast_expr,
|
||||
data_type,
|
||||
..
|
||||
} if matches!(data_type, DataType::Int64 | DataType::Float64) => {
|
||||
if let Some(new_type) = get_data_type_by_alias_name(&data_type.to_string()) {
|
||||
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
|
||||
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
|
||||
(**cast_expr).clone(),
|
||||
new_type.as_arrow_type().to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Type alias
|
||||
Expr::Cast {
|
||||
data_type: DataType::Custom(name, tokens),
|
||||
expr: cast_expr,
|
||||
..
|
||||
} if name.0.len() == 1 && tokens.is_empty() => {
|
||||
if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) {
|
||||
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
|
||||
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
|
||||
(**cast_expr).clone(),
|
||||
new_type.as_arrow_type().to_string(),
|
||||
));
|
||||
}
|
||||
} if get_type_by_alias(data_type).is_some() => {
|
||||
// Safety: checked in the match arm.
|
||||
let new_type = get_type_by_alias(data_type).unwrap();
|
||||
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
|
||||
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
|
||||
(**cast_expr).clone(),
|
||||
new_type.as_arrow_type().to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -153,20 +141,44 @@ impl TransformRule for TypeAliasTransformRule {
|
||||
}
|
||||
|
||||
fn replace_type_alias(data_type: &mut DataType) {
|
||||
match data_type {
|
||||
// TODO(dennis): The sqlparser latest version contains the Int8 alias for Postgres Bigint.
|
||||
// Which means 8 bytes in postgres (not 8 bits). If we upgrade the sqlparser, need to process it.
|
||||
// See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8
|
||||
DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => {
|
||||
if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) {
|
||||
*data_type = new_type;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
if let Some(new_type) = get_type_by_alias(data_type) {
|
||||
*data_type = new_type;
|
||||
}
|
||||
}
|
||||
|
||||
pub fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
|
||||
/// Get data type from alias type.
|
||||
/// Returns the mapped data type if the input data type is an alias that we need to replace.
|
||||
// Remember to update `get_data_type_by_alias_name()` if you modify this method.
|
||||
pub(crate) fn get_type_by_alias(data_type: &DataType) -> Option<DataType> {
|
||||
match data_type {
|
||||
// The sqlparser latest version contains the Int8 alias for Postgres Bigint.
|
||||
// Which means 8 bytes in postgres (not 8 bits).
|
||||
// See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8
|
||||
DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => {
|
||||
get_data_type_by_alias_name(name.0[0].value.as_str())
|
||||
}
|
||||
DataType::Int8(None) => Some(DataType::TinyInt(None)),
|
||||
DataType::Int16 => Some(DataType::SmallInt(None)),
|
||||
DataType::Int32 => Some(DataType::Int(None)),
|
||||
DataType::Int64 => Some(DataType::BigInt(None)),
|
||||
DataType::UInt8 => Some(DataType::UnsignedTinyInt(None)),
|
||||
DataType::UInt16 => Some(DataType::UnsignedSmallInt(None)),
|
||||
DataType::UInt32 => Some(DataType::UnsignedInt(None)),
|
||||
DataType::UInt64 => Some(DataType::UnsignedBigInt(None)),
|
||||
DataType::Float32 => Some(DataType::Float(None)),
|
||||
DataType::Float64 => Some(DataType::Double),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Get the mapped data type from alias name.
|
||||
/// It only supports the following types of alias:
|
||||
/// - timestamps
|
||||
/// - ints
|
||||
/// - floats
|
||||
/// - texts
|
||||
// Remember to update `get_type_alias()` if you modify this method.
|
||||
pub(crate) fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
|
||||
match name.to_uppercase().as_ref() {
|
||||
// Timestamp type alias
|
||||
"TIMESTAMP_S" | "TIMESTAMP_SEC" | "TIMESTAMPSECOND" => {
|
||||
@@ -183,6 +195,7 @@ pub fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
|
||||
Some(DataType::Timestamp(Some(9), TimezoneInfo::None))
|
||||
}
|
||||
// Number type alias
|
||||
// We keep them for backward compatibility.
|
||||
"INT8" => Some(DataType::TinyInt(None)),
|
||||
"INT16" => Some(DataType::SmallInt(None)),
|
||||
"INT32" => Some(DataType::Int(None)),
|
||||
@@ -384,13 +397,13 @@ CREATE TABLE data_types (
|
||||
tt TEXT,
|
||||
mt TEXT,
|
||||
lt TEXT,
|
||||
tint INT8,
|
||||
tint TINYINT,
|
||||
sint SMALLINT,
|
||||
i INT,
|
||||
bint INT64,
|
||||
bint BIGINT,
|
||||
v VARCHAR,
|
||||
f FLOAT,
|
||||
d FLOAT64,
|
||||
d DOUBLE,
|
||||
b BOOLEAN,
|
||||
vb VARBINARY,
|
||||
dt DATE,
|
||||
|
||||
@@ -17,7 +17,7 @@ use std::fmt::{Display, Formatter};
|
||||
|
||||
use sqlparser::ast::{Expr, ObjectName, Query, SetExpr, SqlOption, TableFactor, Value};
|
||||
|
||||
use crate::error::{InvalidTableOptionValueSnafu, Result};
|
||||
use crate::error::{InvalidSqlSnafu, InvalidTableOptionValueSnafu, Result};
|
||||
|
||||
/// Format an [ObjectName] without any quote of its idents.
|
||||
pub fn format_raw_object_name(name: &ObjectName) -> String {
|
||||
@@ -41,7 +41,12 @@ pub fn format_raw_object_name(name: &ObjectName) -> String {
|
||||
}
|
||||
|
||||
pub fn parse_option_string(option: SqlOption) -> Result<(String, String)> {
|
||||
let (key, value) = (option.name, option.value);
|
||||
let SqlOption::KeyValue { key, value } = option else {
|
||||
return InvalidSqlSnafu {
|
||||
msg: "Expecting a key-value pair in the option",
|
||||
}
|
||||
.fail();
|
||||
};
|
||||
let v = match value {
|
||||
Expr::Value(Value::SingleQuotedString(v)) | Expr::Value(Value::DoubleQuotedString(v)) => v,
|
||||
Expr::Identifier(v) => v.value,
|
||||
|
||||
Reference in New Issue
Block a user