feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)

* change dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: adapt to arrow's interval array

* chore: fix compile errors in datatypes crate

* chore: fix api crate compiler errors

* chore: fix compiler errors in common-grpc

* chore: fix common-datasource errors

* chore: fix deprecated code in common-datasource

* fix promql and physical plan related

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip: upgrading network deps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* block on updating `sqlparser`

* upgrade sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adapt new df's trait requirements

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix compiler errors in mito2

* chore: fix common-function crate errors

* chore: fix catalog errors

* change import path

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix some errors in query crate

* chore: fix some errors in query crate

* aggr expr and some other tiny fixes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix expr related errors in query crate

* chore: fix query serializer and admin command

* chore: fix grpc services

* feat: axum serve

* chore: fix http server

* remove handle_error handler
* refactor timeout layer
* serve axum

* chore: fix flow aggr functions

* chore: fix flow

* feat: fix errors in meta-srv

* boxed()
* use TokioIo

* feat!: Remove script crate and python feature (#5321)

* feat: exclude script crate

* chore: simplify feature

* feat: remove the script crate

* chore: remove python feature and some comments

* chore: fix warning

* chore: fix servers tests compiler errors

* feat: fix tests-integration errors

* chore: fix unused

* test: fix catalog test

* chore: fix compiler errors for crates using common-meta

testing feature is enabled when check with --workspace

* test: use display for logical plan test

* test: implement rewrite for ScanHintRule

* fix: http server build panic

* test: fix mito test

* fix: sql parser type alias error

* test: fix TestClient not listen

* test: some flow tests

* test(flow): more fix

* fix: test_otlp_logs

* test: fix promql test that using deprecated method fun()

* fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64

* test: fix infer schema test case

* test: fix tests related to plan display

* chore: fix last flow test

* test: fix function format related assertion

* test: use larger port range for tests

* fix: test_otlp_traces

* fix: test_otlp_metrics

* fix range query and dist plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: flow handle distinct use deprecated field

* fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs

* test: fix deserialize test

* test: reduce split key case num

* tests: lower case aggr func name

* test: fix some sqlness tests

* tests: more sqlness fix

* tests: fixed sqlness test

* commit non-bug changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: make our udf correct

* fix: implement empty methods of ContextProvider for DfContextProviderAdapter

* test: update sqlness test result

* chore: remove unused

* fix: provide alias name for AggregateExprBuilder in range plan

* test: update range query result

* fix: implement missing ContextProvider methods for DfContextProviderAdapter

* test: update timestamps, cte result

* fix: supports empty projection in mito

* test: update comment for cte test

* fix: support projection for numbers

* test: update test cases after projection fix

* fix: fix range select first_value/last_value

* fix: handle CAST and time index conflict

* fix: handle order by correctly in range first_value/last_value

* test: update sqlness result

* test: update view test result

* test: update decimal test

wait for https://github.com/apache/datafusion/pull/14126 to fix this

* feat: remove redundant physical optimization

todo(ruihang): Check if we can remove this.

* test: update sqlness test result

* chore: range select default sort use nulls_first = false

* test: update filter push down test result

* test: comment deciaml test to avoid different panic message

* test: update some distributed test result

* test: update test for distributed count and filter push down

* test: update subqueries test

* fix: SessionState may overwrite our UDFs

* chore: fix compiler errors after merging main

* fix: fix elasticsearch and dashboard router panic

* chore: fix common-functions tests

* chore: update sqlness result

* test: fix id keyword and update sqlness result

* test: fix flow_null test

* fix: enlarge thread size in debug mode to avoid overflow

* chore: fix warnings in common-function

* chore: fix warning in flow

* chore: fix warnings in query crate

* chore: remove unused warnings

* chore: fix deprecated warnings for parquet

* chore: fix deprecated warning in servers crate

* style: fix clippy

* test: enlarge mito cache tttl test ttl time

* chore: fix typo

* style: fmt toml

* refactor: reimplement PartialOrd for RangeSelect

* chore: remove script crate files introduced by merge

* fix: return error if sql option is not kv

* chore: do not use ..default::default()

* chore: per review

* chore: update error message in BuildAdminFunctionArgsSnafu

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* refactor: typed precision

* update sqlness view case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: flow per review

* chore: add example in comment

* chore: warn if parquet stats of timestamp is not INT64

* style: add a newline before derive to make the comment more clear

* test: update sqlness result

* fix: flow from substrait

* chore: change update_range_context log to debug level

* chore: move axum-extra axum-macros to workspace

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
This commit is contained in:
Yingwen
2025-01-23 14:15:40 +08:00
committed by GitHub
parent 3ed085459c
commit 35b635f639
293 changed files with 4067 additions and 3225 deletions

View File

@@ -14,6 +14,7 @@
pub use sqlparser::ast::{
visit_expressions_mut, visit_statements_mut, BinaryOperator, ColumnDef, ColumnOption,
ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName,
SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor, VisitorMut,
ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArguments,
Ident, ObjectName, SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor,
VisitorMut,
};

View File

@@ -72,7 +72,7 @@ pub enum Error {
},
// Syntax error from sql parser.
#[snafu(display(""))]
#[snafu(display("Invalid SQL syntax"))]
Syntax {
#[snafu(source)]
error: ParserError,
@@ -81,7 +81,7 @@ pub enum Error {
},
// Syntax error from tql parser.
#[snafu(display(""))]
#[snafu(display("Invalid TQL syntax"))]
TQLSyntax {
#[snafu(source)]
error: TQLError,

View File

@@ -14,6 +14,7 @@
#![feature(box_patterns)]
#![feature(assert_matches)]
#![feature(if_let_guard)]
#![feature(let_chains)]
pub mod ast;

View File

@@ -50,7 +50,7 @@ impl ParserContext<'_> {
/// Parses parser context to Query.
pub fn parser_query(&mut self) -> Result<Box<Query>> {
Ok(Box::new(self.parser.parse_query().context(SyntaxSnafu)?))
self.parser.parse_query().context(SyntaxSnafu)
}
/// Parses SQL with given dialect

View File

@@ -46,6 +46,8 @@ impl ParserContext<'_> {
#[cfg(test)]
mod tests {
use sqlparser::ast::FunctionArguments;
use super::*;
use crate::ast::{Expr, Function, FunctionArg, FunctionArgExpr, Value};
use crate::dialect::GreptimeDbDialect;
@@ -62,9 +64,12 @@ mod tests {
let stmt = result.remove(0);
match &stmt {
Statement::Admin(Admin::Func(Function { name, args, .. })) => {
let FunctionArguments::List(arg_list) = args else {
unreachable!()
};
assert_eq!("flush_table", name.to_string());
assert_eq!(args.len(), 1);
assert!(matches!(&args[0],
assert_eq!(arg_list.args.len(), 1);
assert!(matches!(&arg_list.args[0],
FunctionArg::Unnamed(FunctionArgExpr::Expr(
Expr::Value(Value::SingleQuotedString(s))
)) if s == "test"));
@@ -86,8 +91,11 @@ mod tests {
let stmt = result.remove(0);
match &stmt {
Statement::Admin(Admin::Func(Function { name, args, .. })) => {
let FunctionArguments::List(arg_list) = args else {
unreachable!()
};
assert_eq!("test", name.to_string());
assert_eq!(args.len(), 0);
assert_eq!(arg_list.args.len(), 0);
}
_ => unreachable!(),
}

View File

@@ -644,7 +644,7 @@ mod tests {
let err = result.output_msg();
assert_eq!(
err,
"sql parser error: Expected COLUMN, found: a at Line: 1, Column 30"
"Invalid SQL syntax: sql parser error: Expected: COLUMN, found: a at Line: 1, Column: 30"
);
let sql = "ALTER TABLE my_metric_1 DROP COLUMN a";
@@ -790,7 +790,7 @@ mod tests {
ParserContext::create_with_dialect(sql, &GreptimeDbDialect {}, ParseOptions::default())
.unwrap_err();
let err = result.output_msg();
assert_eq!(err, "sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t");
assert_eq!(err, "Invalid SQL syntax: sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t");
let sql = "ALTER TABLE test_table RENAME table_t";
let mut result =

View File

@@ -44,9 +44,8 @@ use crate::statements::create::{
CreateTableLike, CreateView, Partitions, TableConstraint, VECTOR_OPT_DIM,
};
use crate::statements::statement::Statement;
use crate::statements::{
get_data_type_by_alias_name, sql_data_type_to_concrete_data_type, OptionMap,
};
use crate::statements::transform::type_alias::get_data_type_by_alias_name;
use crate::statements::{sql_data_type_to_concrete_data_type, OptionMap};
use crate::util::parse_option_string;
pub const ENGINE: &str = "ENGINE";
@@ -291,13 +290,7 @@ impl<'a> ParserContext<'a> {
reason: format!("cannot cast {} to interval type", expire_after_expr),
})?;
if let ScalarValue::IntervalMonthDayNano(Some(nanoseconds)) = expire_after_lit {
Some(
i64::try_from(nanoseconds / 1_000_000_000)
.ok()
.with_context(|| InvalidIntervalSnafu {
reason: format!("interval {} overflows", nanoseconds),
})?,
)
Some(nanoseconds.nanoseconds / 1_000_000_000)
} else {
unreachable!()
}
@@ -326,7 +319,7 @@ impl<'a> ParserContext<'a> {
.expect_keyword(Keyword::AS)
.context(SyntaxSnafu)?;
let query = Box::new(self.parser.parse_query().context(error::SyntaxSnafu)?);
let query = self.parser.parse_query().context(error::SyntaxSnafu)?;
Ok(Statement::CreateFlow(CreateFlow {
flow_name,
@@ -1179,7 +1172,7 @@ mod tests {
assert_column_def(&columns[0].column_def, "host", "STRING");
assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP");
assert_column_def(&columns[2].column_def, "cpu", "FLOAT");
assert_column_def(&columns[3].column_def, "memory", "FLOAT64");
assert_column_def(&columns[3].column_def, "memory", "DOUBLE");
let constraints = &c.constraints;
assert_eq!(
@@ -1758,7 +1751,7 @@ ENGINE=mito";
assert!(result
.unwrap_err()
.output_msg()
.contains("sql parser error: Expected ON, found: COLUMNS"));
.contains("sql parser error: Expected: ON, found: COLUMNS"));
}
#[test]
@@ -1835,7 +1828,7 @@ ENGINE=mito";
assert_column_def(&columns[0].column_def, "host", "STRING");
assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP");
assert_column_def(&columns[2].column_def, "cpu", "FLOAT");
assert_column_def(&columns[3].column_def, "memory", "FLOAT64");
assert_column_def(&columns[3].column_def, "memory", "DOUBLE");
let constraints = &c.constraints;
assert_eq!(

View File

@@ -68,12 +68,13 @@ mod tests {
with_hints: vec![],
partitions: vec![],
version: None,
with_ordinality: false,
},
joins: vec![],
}],
lateral_views: vec![],
selection: None,
group_by: GroupByExpr::Expressions(vec![]),
group_by: GroupByExpr::Expressions(vec![], vec![]),
cluster_by: vec![],
distribute_by: vec![],
sort_by: vec![],
@@ -81,18 +82,24 @@ mod tests {
qualify: None,
named_window: vec![],
value_table_mode: None,
top_before_distinct: false,
prewhere: None,
window_before_qualify: false,
connect_by: None,
};
let sp_statement = SpStatement::Query(Box::new(SpQuery {
with: None,
body: Box::new(sqlparser::ast::SetExpr::Select(Box::new(select))),
order_by: vec![],
order_by: None,
limit: None,
limit_by: vec![],
offset: None,
fetch: None,
locks: vec![],
for_clause: None,
settings: None,
format_clause: None,
}));
let explain = Explain::try_from(SpStatement::Explain {
@@ -101,6 +108,8 @@ mod tests {
verbose: false,
statement: Box::new(sp_statement),
format: None,
query_plan: false,
options: None,
})
.unwrap();

View File

@@ -24,7 +24,7 @@ impl ParserContext<'_> {
pub(crate) fn parse_query(&mut self) -> Result<Statement> {
let spquery = self.parser.parse_query().context(error::SyntaxSnafu)?;
Ok(Statement::Query(Box::new(Query::try_from(spquery)?)))
Ok(Statement::Query(Box::new(Query::try_from(*spquery)?)))
}
}
@@ -56,6 +56,6 @@ mod tests {
assert!(result
.unwrap_err()
.output_msg()
.contains("Expected an expression"));
.contains("Expected: an expression"));
}
}

View File

@@ -28,11 +28,14 @@ impl ParserContext<'_> {
let spstatement = self.parser.parse_set().context(error::SyntaxSnafu)?;
match spstatement {
SpStatement::SetVariable {
variable,
variables,
value,
hivevar,
..
} if !hivevar => Ok(Statement::SetVariables(SetVariables { variable, value })),
} if !hivevar => Ok(Statement::SetVariables(SetVariables {
variable: (*variables)[0].clone(),
value,
})),
SpStatement::SetTimeZone { value, .. } => Ok(Statement::SetVariables(SetVariables {
variable: ObjectName(vec![Ident {

View File

@@ -18,6 +18,7 @@ use chrono::Utc;
use datafusion::config::ConfigOptions;
use datafusion::error::Result as DfResult;
use datafusion::execution::context::SessionState;
use datafusion::execution::SessionStateBuilder;
use datafusion::optimizer::simplify_expressions::ExprSimplifier;
use datafusion_common::{DFSchema, ScalarValue};
use datafusion_expr::execution_props::ExecutionProps;
@@ -72,7 +73,11 @@ struct StubContextProvider {
impl Default for StubContextProvider {
fn default() -> Self {
Self {
state: SessionState::new_with_config_rt(Default::default(), Default::default()),
state: SessionStateBuilder::new()
.with_config(Default::default())
.with_runtime_env(Default::default())
.with_default_features()
.build(),
}
}
}

View File

@@ -28,7 +28,7 @@ pub mod set_variables;
pub mod show;
pub mod statement;
pub mod tql;
mod transform;
pub(crate) mod transform;
pub mod truncate;
use std::str::FromStr;
@@ -61,7 +61,7 @@ use crate::error::{
};
use crate::statements::create::Column;
pub use crate::statements::option_map::OptionMap;
pub use crate::statements::transform::{get_data_type_by_alias_name, transform_statements};
pub(crate) use crate::statements::transform::transform_statements;
const VECTOR_TYPE_NAME: &str = "VECTOR";

View File

@@ -695,8 +695,8 @@ CREATE TABLE t2 LIKE t1"#,
CREATE EXTERNAL TABLE city (
host STRING,
ts TIMESTAMP,
cpu FLOAT64 DEFAULT 0,
memory FLOAT64,
cpu DOUBLE DEFAULT 0,
memory DOUBLE,
TIME INDEX (ts),
PRIMARY KEY (host)
)

View File

@@ -13,7 +13,9 @@
// limitations under the License.
use serde::Serialize;
use sqlparser::ast::{ObjectName, Query, SetExpr, Statement, UnaryOperator, Values};
use sqlparser::ast::{
Insert as SpInsert, ObjectName, Query, SetExpr, Statement, UnaryOperator, Values,
};
use sqlparser::parser::ParserError;
use sqlparser_derive::{Visit, VisitMut};
@@ -39,14 +41,14 @@ macro_rules! parse_fail {
impl Insert {
pub fn table_name(&self) -> &ObjectName {
match &self.inner {
Statement::Insert { table_name, .. } => table_name,
Statement::Insert(insert) => &insert.table_name,
_ => unreachable!(),
}
}
pub fn columns(&self) -> Vec<&String> {
match &self.inner {
Statement::Insert { columns, .. } => columns.iter().map(|ident| &ident.value).collect(),
Statement::Insert(insert) => insert.columns.iter().map(|ident| &ident.value).collect(),
_ => unreachable!(),
}
}
@@ -54,14 +56,14 @@ impl Insert {
/// Extracts the literal insert statement body if possible
pub fn values_body(&self) -> Result<Vec<Vec<Value>>> {
match &self.inner {
Statement::Insert {
Statement::Insert(SpInsert {
source:
Some(box Query {
body: box SetExpr::Values(Values { rows, .. }),
..
}),
..
} => sql_exprs_to_values(rows),
}) => sql_exprs_to_values(rows),
_ => unreachable!(),
}
}
@@ -70,14 +72,14 @@ impl Insert {
/// The rules is the same as function `values_body()`.
pub fn can_extract_values(&self) -> bool {
match &self.inner {
Statement::Insert {
Statement::Insert(SpInsert {
source:
Some(box Query {
body: box SetExpr::Values(Values { rows, .. }),
..
}),
..
} => rows.iter().all(|es| {
}) => rows.iter().all(|es| {
es.iter().all(|expr| match expr {
Expr::Value(_) => true,
Expr::Identifier(ident) => {
@@ -100,10 +102,10 @@ impl Insert {
pub fn query_body(&self) -> Result<Option<GtQuery>> {
Ok(match &self.inner {
Statement::Insert {
Statement::Insert(SpInsert {
source: Some(box query),
..
} => Some(query.clone().try_into()?),
}) => Some(query.clone().try_into()?),
_ => None,
})
}

View File

@@ -12,20 +12,19 @@
// See the License for the specific language governing permissions and
// limitations under the License.
mod expand_interval;
pub(crate) mod type_alias;
use std::ops::ControlFlow;
use std::sync::Arc;
use expand_interval::ExpandIntervalTransformRule;
use lazy_static::lazy_static;
use sqlparser::ast::{visit_expressions_mut, Expr};
use type_alias::TypeAliasTransformRule;
use crate::error::Result;
use crate::statements::statement::Statement;
mod expand_interval;
mod type_alias;
use expand_interval::ExpandIntervalTransformRule;
pub use type_alias::get_data_type_by_alias_name;
use type_alias::TypeAliasTransformRule;
lazy_static! {
/// [TransformRule] registry

View File

@@ -371,6 +371,7 @@ mod tests {
expr: single_quoted_string_expr("3y2mon".to_string()),
data_type: DataType::Interval,
format: None,
kind: sqlparser::ast::CastKind::Cast,
};
let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_interval_expr);
@@ -393,6 +394,7 @@ mod tests {
expr: single_quoted_string_expr("5".to_string()),
data_type: DataType::Int64,
format: None,
kind: sqlparser::ast::CastKind::Cast,
};
let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_i64_expr);
assert_eq!(control_flow, ControlFlow::Continue(()));
@@ -402,6 +404,7 @@ mod tests {
expr: single_quoted_string_expr("5".to_string()),
data_type: DataType::Int64,
format: None,
kind: sqlparser::ast::CastKind::Cast,
}
);
}

View File

@@ -16,7 +16,8 @@ use std::ops::ControlFlow;
use datatypes::data_type::DataType as GreptimeDataType;
use sqlparser::ast::{
DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName, Value,
DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArgumentList, Ident,
ObjectName, Value,
};
use crate::error::Result;
@@ -75,18 +76,21 @@ impl TransformRule for TypeAliasTransformRule {
fn cast_expr_to_arrow_cast_func(expr: Expr, cast_type: String) -> Function {
Function {
name: ObjectName(vec![Ident::new("arrow_cast")]),
args: vec![
FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)),
FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
Value::SingleQuotedString(cast_type),
))),
],
args: sqlparser::ast::FunctionArguments::List(FunctionArgumentList {
args: vec![
FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)),
FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value(
Value::SingleQuotedString(cast_type),
))),
],
duplicate_treatment: None,
clauses: vec![],
}),
filter: None,
null_treatment: None,
over: None,
distinct: false,
special: false,
order_by: vec![],
parameters: sqlparser::ast::FunctionArguments::None,
within_group: vec![],
}
}
@@ -99,30 +103,14 @@ impl TransformRule for TypeAliasTransformRule {
expr: cast_expr,
data_type,
..
} if matches!(data_type, DataType::Int64 | DataType::Float64) => {
if let Some(new_type) = get_data_type_by_alias_name(&data_type.to_string()) {
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
(**cast_expr).clone(),
new_type.as_arrow_type().to_string(),
));
}
}
}
// Type alias
Expr::Cast {
data_type: DataType::Custom(name, tokens),
expr: cast_expr,
..
} if name.0.len() == 1 && tokens.is_empty() => {
if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) {
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
(**cast_expr).clone(),
new_type.as_arrow_type().to_string(),
));
}
} if get_type_by_alias(data_type).is_some() => {
// Safety: checked in the match arm.
let new_type = get_type_by_alias(data_type).unwrap();
if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) {
*expr = Expr::Function(cast_expr_to_arrow_cast_func(
(**cast_expr).clone(),
new_type.as_arrow_type().to_string(),
));
}
}
@@ -153,20 +141,44 @@ impl TransformRule for TypeAliasTransformRule {
}
fn replace_type_alias(data_type: &mut DataType) {
match data_type {
// TODO(dennis): The sqlparser latest version contains the Int8 alias for Postgres Bigint.
// Which means 8 bytes in postgres (not 8 bits). If we upgrade the sqlparser, need to process it.
// See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8
DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => {
if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) {
*data_type = new_type;
}
}
_ => {}
if let Some(new_type) = get_type_by_alias(data_type) {
*data_type = new_type;
}
}
pub fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
/// Get data type from alias type.
/// Returns the mapped data type if the input data type is an alias that we need to replace.
// Remember to update `get_data_type_by_alias_name()` if you modify this method.
pub(crate) fn get_type_by_alias(data_type: &DataType) -> Option<DataType> {
match data_type {
// The sqlparser latest version contains the Int8 alias for Postgres Bigint.
// Which means 8 bytes in postgres (not 8 bits).
// See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8
DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => {
get_data_type_by_alias_name(name.0[0].value.as_str())
}
DataType::Int8(None) => Some(DataType::TinyInt(None)),
DataType::Int16 => Some(DataType::SmallInt(None)),
DataType::Int32 => Some(DataType::Int(None)),
DataType::Int64 => Some(DataType::BigInt(None)),
DataType::UInt8 => Some(DataType::UnsignedTinyInt(None)),
DataType::UInt16 => Some(DataType::UnsignedSmallInt(None)),
DataType::UInt32 => Some(DataType::UnsignedInt(None)),
DataType::UInt64 => Some(DataType::UnsignedBigInt(None)),
DataType::Float32 => Some(DataType::Float(None)),
DataType::Float64 => Some(DataType::Double),
_ => None,
}
}
/// Get the mapped data type from alias name.
/// It only supports the following types of alias:
/// - timestamps
/// - ints
/// - floats
/// - texts
// Remember to update `get_type_alias()` if you modify this method.
pub(crate) fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
match name.to_uppercase().as_ref() {
// Timestamp type alias
"TIMESTAMP_S" | "TIMESTAMP_SEC" | "TIMESTAMPSECOND" => {
@@ -183,6 +195,7 @@ pub fn get_data_type_by_alias_name(name: &str) -> Option<DataType> {
Some(DataType::Timestamp(Some(9), TimezoneInfo::None))
}
// Number type alias
// We keep them for backward compatibility.
"INT8" => Some(DataType::TinyInt(None)),
"INT16" => Some(DataType::SmallInt(None)),
"INT32" => Some(DataType::Int(None)),
@@ -384,13 +397,13 @@ CREATE TABLE data_types (
tt TEXT,
mt TEXT,
lt TEXT,
tint INT8,
tint TINYINT,
sint SMALLINT,
i INT,
bint INT64,
bint BIGINT,
v VARCHAR,
f FLOAT,
d FLOAT64,
d DOUBLE,
b BOOLEAN,
vb VARBINARY,
dt DATE,

View File

@@ -17,7 +17,7 @@ use std::fmt::{Display, Formatter};
use sqlparser::ast::{Expr, ObjectName, Query, SetExpr, SqlOption, TableFactor, Value};
use crate::error::{InvalidTableOptionValueSnafu, Result};
use crate::error::{InvalidSqlSnafu, InvalidTableOptionValueSnafu, Result};
/// Format an [ObjectName] without any quote of its idents.
pub fn format_raw_object_name(name: &ObjectName) -> String {
@@ -41,7 +41,12 @@ pub fn format_raw_object_name(name: &ObjectName) -> String {
}
pub fn parse_option_string(option: SqlOption) -> Result<(String, String)> {
let (key, value) = (option.name, option.value);
let SqlOption::KeyValue { key, value } = option else {
return InvalidSqlSnafu {
msg: "Expecting a key-value pair in the option",
}
.fail();
};
let v = match value {
Expr::Value(Value::SingleQuotedString(v)) | Expr::Value(Value::DoubleQuotedString(v)) => v,
Expr::Identifier(v) => v.value,