mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-08-12 09:19:44 +00:00
feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: upgrading network deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adapt new df's trait requirements Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi <jiachun_feng@proton.me> * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me>
This commit is contained in:
Yingwen
GitHub
@@ -31,7 +31,7 @@ derive_builder.workspace = true
|
||||
futures.workspace = true
|
||||
lazy_static.workspace = true
|
||||
object-store.workspace = true
|
||||
orc-rust = { git = "https://github.com/datafusion-contrib/datafusion-orc.git", rev = "502217315726314c4008808fe169764529640599", default-features = false, features = [
|
||||
orc-rust = { version = "0.5", default-features = false, features = [
|
||||
"async",
|
||||
] }
|
||||
parquet.workspace = true
|
||||
|
||||
@@ -126,8 +126,7 @@ impl ArrowDecoder for arrow::csv::reader::Decoder {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(deprecated)]
|
||||
impl ArrowDecoder for arrow::json::RawDecoder {
|
||||
impl ArrowDecoder for arrow::json::reader::Decoder {
|
||||
fn decode(&mut self, buf: &[u8]) -> result::Result<usize, ArrowError> {
|
||||
self.decode(buf)
|
||||
}
|
||||
|
||||
@@ -17,8 +17,7 @@ use std::str::FromStr;
|
||||
use std::sync::Arc;
|
||||
|
||||
use arrow::csv;
|
||||
#[allow(deprecated)]
|
||||
use arrow::csv::reader::infer_reader_schema as infer_csv_schema;
|
||||
use arrow::csv::reader::Format;
|
||||
use arrow::record_batch::RecordBatch;
|
||||
use arrow_schema::{Schema, SchemaRef};
|
||||
use async_trait::async_trait;
|
||||
@@ -161,7 +160,6 @@ impl FileOpener for CsvOpener {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(deprecated)]
|
||||
#[async_trait]
|
||||
impl FileFormat for CsvFormat {
|
||||
async fn infer_schema(&self, store: &ObjectStore, path: &str) -> Result<Schema> {
|
||||
@@ -188,9 +186,12 @@ impl FileFormat for CsvFormat {
|
||||
common_runtime::spawn_blocking_global(move || {
|
||||
let reader = SyncIoBridge::new(decoded);
|
||||
|
||||
let (schema, _records_read) =
|
||||
infer_csv_schema(reader, delimiter, schema_infer_max_record, has_header)
|
||||
.context(error::InferSchemaSnafu)?;
|
||||
let format = Format::default()
|
||||
.with_delimiter(delimiter)
|
||||
.with_header(has_header);
|
||||
let (schema, _records_read) = format
|
||||
.infer_schema(reader, schema_infer_max_record)
|
||||
.context(error::InferSchemaSnafu)?;
|
||||
Ok(schema)
|
||||
})
|
||||
.await
|
||||
@@ -253,7 +254,7 @@ mod tests {
|
||||
"c7: Int64: NULL",
|
||||
"c8: Int64: NULL",
|
||||
"c9: Int64: NULL",
|
||||
"c10: Int64: NULL",
|
||||
"c10: Utf8: NULL",
|
||||
"c11: Float64: NULL",
|
||||
"c12: Float64: NULL",
|
||||
"c13: Utf8: NULL"
|
||||
|
||||
@@ -20,8 +20,7 @@ use std::sync::Arc;
|
||||
use arrow::datatypes::SchemaRef;
|
||||
use arrow::json::reader::{infer_json_schema_from_iterator, ValueIter};
|
||||
use arrow::json::writer::LineDelimited;
|
||||
#[allow(deprecated)]
|
||||
use arrow::json::{self, RawReaderBuilder};
|
||||
use arrow::json::{self, ReaderBuilder};
|
||||
use arrow::record_batch::RecordBatch;
|
||||
use arrow_schema::Schema;
|
||||
use async_trait::async_trait;
|
||||
@@ -140,7 +139,6 @@ impl JsonOpener {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(deprecated)]
|
||||
impl FileOpener for JsonOpener {
|
||||
fn open(&self, meta: FileMeta) -> DataFusionResult<FileOpenFuture> {
|
||||
open_with_decoder(
|
||||
@@ -148,7 +146,7 @@ impl FileOpener for JsonOpener {
|
||||
meta.location().to_string(),
|
||||
self.compression_type,
|
||||
|| {
|
||||
RawReaderBuilder::new(self.projected_schema.clone())
|
||||
ReaderBuilder::new(self.projected_schema.clone())
|
||||
.with_batch_size(self.batch_size)
|
||||
.build_decoder()
|
||||
.map_err(DataFusionError::from)
|
||||
|
||||
@@ -91,6 +91,7 @@ mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prelude::{TypeSignature, Volatility};
|
||||
use datatypes::arrow::datatypes::IntervalDayTime;
|
||||
use datatypes::prelude::ConcreteDataType;
|
||||
use datatypes::value::Value;
|
||||
use datatypes::vectors::{
|
||||
@@ -134,7 +135,12 @@ mod tests {
|
||||
|
||||
let times = vec![Some(123), None, Some(42), None];
|
||||
// Intervals in milliseconds
|
||||
let intervals = vec![1000, 2000, 3000, 1000];
|
||||
let intervals = vec![
|
||||
IntervalDayTime::new(0, 1000),
|
||||
IntervalDayTime::new(0, 2000),
|
||||
IntervalDayTime::new(0, 3000),
|
||||
IntervalDayTime::new(0, 1000),
|
||||
];
|
||||
let results = [Some(124), None, Some(45), None];
|
||||
|
||||
let time_vector = TimestampSecondVector::from(times.clone());
|
||||
|
||||
@@ -91,6 +91,7 @@ mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prelude::{TypeSignature, Volatility};
|
||||
use datatypes::arrow::datatypes::IntervalDayTime;
|
||||
use datatypes::prelude::ConcreteDataType;
|
||||
use datatypes::value::Value;
|
||||
use datatypes::vectors::{
|
||||
@@ -139,7 +140,12 @@ mod tests {
|
||||
|
||||
let times = vec![Some(123), None, Some(42), None];
|
||||
// Intervals in milliseconds
|
||||
let intervals = vec![1000, 2000, 3000, 1000];
|
||||
let intervals = vec![
|
||||
IntervalDayTime::new(0, 1000),
|
||||
IntervalDayTime::new(0, 2000),
|
||||
IntervalDayTime::new(0, 3000),
|
||||
IntervalDayTime::new(0, 1000),
|
||||
];
|
||||
let results = [Some(122), None, Some(39), None];
|
||||
|
||||
let time_vector = TimestampSecondVector::from(times.clone());
|
||||
|
||||
@@ -21,10 +21,9 @@ use common_query::error::{
|
||||
};
|
||||
use datafusion::common::tree_node::{Transformed, TreeNode, TreeNodeIterator, TreeNodeRecursion};
|
||||
use datafusion::common::{DFSchema, Result as DfResult};
|
||||
use datafusion::execution::context::SessionState;
|
||||
use datafusion::execution::SessionStateBuilder;
|
||||
use datafusion::logical_expr::{self, Expr, Volatility};
|
||||
use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner};
|
||||
use datafusion::prelude::SessionConfig;
|
||||
use datatypes::arrow::array::RecordBatch;
|
||||
use datatypes::arrow::datatypes::{DataType, Field};
|
||||
use datatypes::prelude::VectorRef;
|
||||
@@ -104,8 +103,7 @@ impl MatchesFunction {
|
||||
let like_expr = ast.into_like_expr(col_name);
|
||||
|
||||
let input_schema = Self::input_schema();
|
||||
let session_state =
|
||||
SessionState::new_with_config_rt(SessionConfig::default(), Arc::default());
|
||||
let session_state = SessionStateBuilder::new().with_default_features().build();
|
||||
let planner = DefaultPhysicalPlanner::default();
|
||||
let physical_expr = planner
|
||||
.create_physical_expr(&like_expr, &input_schema, &session_state)
|
||||
@@ -131,7 +129,7 @@ impl MatchesFunction {
|
||||
}
|
||||
|
||||
fn input_schema() -> DFSchema {
|
||||
DFSchema::from_unqualifed_fields(
|
||||
DFSchema::from_unqualified_fields(
|
||||
[Arc::new(Field::new("data", DataType::Utf8, true))].into(),
|
||||
HashMap::new(),
|
||||
)
|
||||
|
||||
@@ -42,7 +42,7 @@ impl Function for BuildFunction {
|
||||
}
|
||||
|
||||
fn signature(&self) -> Signature {
|
||||
Signature::uniform(0, vec![], Volatility::Immutable)
|
||||
Signature::nullary(Volatility::Immutable)
|
||||
}
|
||||
|
||||
fn eval(&self, _func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result<VectorRef> {
|
||||
@@ -56,8 +56,6 @@ impl Function for BuildFunction {
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prelude::TypeSignature;
|
||||
|
||||
use super::*;
|
||||
#[test]
|
||||
fn test_build_function() {
|
||||
@@ -67,12 +65,7 @@ mod tests {
|
||||
ConcreteDataType::string_datatype(),
|
||||
build.return_type(&[]).unwrap()
|
||||
);
|
||||
assert!(matches!(build.signature(),
|
||||
Signature {
|
||||
type_signature: TypeSignature::Uniform(0, valid_types),
|
||||
volatility: Volatility::Immutable
|
||||
} if valid_types.is_empty()
|
||||
));
|
||||
assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable));
|
||||
let build_info = common_version::build_info().to_string();
|
||||
let vector = build.eval(FunctionContext::default(), &[]).unwrap();
|
||||
let expect: VectorRef = Arc::new(StringVector::from(vec![build_info]));
|
||||
|
||||
@@ -44,7 +44,7 @@ impl Function for DatabaseFunction {
|
||||
}
|
||||
|
||||
fn signature(&self) -> Signature {
|
||||
Signature::uniform(0, vec![], Volatility::Immutable)
|
||||
Signature::nullary(Volatility::Immutable)
|
||||
}
|
||||
|
||||
fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result<VectorRef> {
|
||||
@@ -116,7 +116,6 @@ impl fmt::Display for SessionUserFunction {
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prelude::TypeSignature;
|
||||
use session::context::QueryContextBuilder;
|
||||
|
||||
use super::*;
|
||||
@@ -128,12 +127,7 @@ mod tests {
|
||||
ConcreteDataType::string_datatype(),
|
||||
build.return_type(&[]).unwrap()
|
||||
);
|
||||
assert!(matches!(build.signature(),
|
||||
Signature {
|
||||
type_signature: TypeSignature::Uniform(0, valid_types),
|
||||
volatility: Volatility::Immutable
|
||||
} if valid_types == vec![]
|
||||
));
|
||||
assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable));
|
||||
|
||||
let query_ctx = QueryContextBuilder::default()
|
||||
.current_schema("test_db".to_string())
|
||||
|
||||
@@ -38,7 +38,7 @@ impl Function for TimezoneFunction {
|
||||
}
|
||||
|
||||
fn signature(&self) -> Signature {
|
||||
Signature::uniform(0, vec![], Volatility::Immutable)
|
||||
Signature::nullary(Volatility::Immutable)
|
||||
}
|
||||
|
||||
fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result<VectorRef> {
|
||||
@@ -58,7 +58,6 @@ impl fmt::Display for TimezoneFunction {
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use common_query::prelude::TypeSignature;
|
||||
use session::context::QueryContextBuilder;
|
||||
|
||||
use super::*;
|
||||
@@ -70,12 +69,7 @@ mod tests {
|
||||
ConcreteDataType::string_datatype(),
|
||||
build.return_type(&[]).unwrap()
|
||||
);
|
||||
assert!(matches!(build.signature(),
|
||||
Signature {
|
||||
type_signature: TypeSignature::Uniform(0, valid_types),
|
||||
volatility: Volatility::Immutable
|
||||
} if valid_types == vec![]
|
||||
));
|
||||
assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable));
|
||||
|
||||
let query_ctx = QueryContextBuilder::default().build().into();
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ impl Function for VersionFunction {
|
||||
}
|
||||
|
||||
fn signature(&self) -> Signature {
|
||||
Signature::exact(vec![], Volatility::Immutable)
|
||||
Signature::nullary(Volatility::Immutable)
|
||||
}
|
||||
|
||||
fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result<VectorRef> {
|
||||
|
||||
@@ -107,7 +107,7 @@ pub enum Error {
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
#[snafu(source)]
|
||||
error: prost::DecodeError,
|
||||
error: prost::UnknownEnumValue,
|
||||
},
|
||||
|
||||
#[snafu(display(
|
||||
@@ -137,7 +137,7 @@ pub enum Error {
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
#[snafu(source)]
|
||||
error: prost::DecodeError,
|
||||
error: prost::UnknownEnumValue,
|
||||
},
|
||||
|
||||
#[snafu(display("Missing alter index options"))]
|
||||
|
||||
@@ -255,7 +255,7 @@ mod tests {
|
||||
ConcreteDataType::from(
|
||||
ColumnDataTypeWrapper::try_new(
|
||||
decimal_column.data_type,
|
||||
decimal_column.datatype_extension.clone(),
|
||||
decimal_column.datatype_extension,
|
||||
)
|
||||
.unwrap()
|
||||
)
|
||||
@@ -351,7 +351,6 @@ mod tests {
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.datatype_extension
|
||||
.clone()
|
||||
)
|
||||
.unwrap()
|
||||
)
|
||||
|
||||
@@ -166,7 +166,7 @@ pub fn build_create_table_expr(
|
||||
default_constraint: vec![],
|
||||
semantic_type,
|
||||
comment: String::new(),
|
||||
datatype_extension: datatype_extension.clone(),
|
||||
datatype_extension: *datatype_extension,
|
||||
options: options.clone(),
|
||||
});
|
||||
}
|
||||
@@ -208,7 +208,7 @@ pub fn extract_new_columns(
|
||||
default_constraint: vec![],
|
||||
semantic_type: expr.semantic_type,
|
||||
comment: String::new(),
|
||||
datatype_extension: expr.datatype_extension.clone(),
|
||||
datatype_extension: *expr.datatype_extension,
|
||||
options: expr.options.clone(),
|
||||
});
|
||||
AddColumn {
|
||||
|
||||
@@ -19,7 +19,8 @@ common-telemetry.workspace = true
|
||||
common-time.workspace = true
|
||||
dashmap.workspace = true
|
||||
datatypes.workspace = true
|
||||
flatbuffers = "23.1"
|
||||
flatbuffers = "24"
|
||||
hyper.workspace = true
|
||||
lazy_static.workspace = true
|
||||
prost.workspace = true
|
||||
snafu.workspace = true
|
||||
@@ -29,6 +30,7 @@ tower.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
criterion = "0.4"
|
||||
hyper-util = { workspace = true, features = ["tokio"] }
|
||||
rand.workspace = true
|
||||
|
||||
[[bench]]
|
||||
|
||||
@@ -25,7 +25,7 @@ use snafu::{OptionExt, ResultExt};
|
||||
use tonic::transport::{
|
||||
Certificate, Channel as InnerChannel, ClientTlsConfig, Endpoint, Identity, Uri,
|
||||
};
|
||||
use tower::make::MakeConnection;
|
||||
use tower::Service;
|
||||
|
||||
use crate::error::{CreateChannelSnafu, InvalidConfigFilePathSnafu, InvalidTlsConfigSnafu, Result};
|
||||
|
||||
@@ -137,8 +137,8 @@ impl ChannelManager {
|
||||
connector: C,
|
||||
) -> Result<InnerChannel>
|
||||
where
|
||||
C: MakeConnection<Uri> + Send + 'static,
|
||||
C::Connection: Unpin + Send + 'static,
|
||||
C: Service<Uri> + Send + 'static,
|
||||
C::Response: hyper::rt::Read + hyper::rt::Write + Send + Unpin,
|
||||
C::Future: Send + 'static,
|
||||
Box<dyn std::error::Error + Send + Sync>: From<C::Error> + Send + 'static,
|
||||
{
|
||||
@@ -607,7 +607,7 @@ mod tests {
|
||||
});
|
||||
|
||||
let (client, _) = tokio::io::duplex(1024);
|
||||
let mut client = Some(client);
|
||||
let mut client = Some(hyper_util::rt::TokioIo::new(client));
|
||||
let res = mgr.reset_with_connector(
|
||||
addr,
|
||||
service_fn(move |_| {
|
||||
|
||||
@@ -205,7 +205,7 @@ pub fn values(arrays: &[VectorRef]) -> Result<Values> {
|
||||
ConcreteDataType::Interval(IntervalType::DayTime(_)),
|
||||
IntervalDayTimeVector,
|
||||
interval_day_time_values,
|
||||
|x| { x.into_native() }
|
||||
|x| { x.to_i64() }
|
||||
),
|
||||
(
|
||||
ConcreteDataType::Interval(IntervalType::MonthDayNano(_)),
|
||||
@@ -232,6 +232,8 @@ pub fn values(arrays: &[VectorRef]) -> Result<Values> {
|
||||
mod tests {
|
||||
use std::sync::Arc;
|
||||
|
||||
use datatypes::arrow::datatypes::{IntervalDayTime, IntervalMonthDayNano};
|
||||
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
@@ -266,7 +268,12 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_convert_arrow_array_interval_day_time() {
|
||||
let array = IntervalDayTimeVector::from(vec![Some(1), Some(2), None, Some(3)]);
|
||||
let array = IntervalDayTimeVector::from(vec![
|
||||
Some(IntervalDayTime::new(0, 1)),
|
||||
Some(IntervalDayTime::new(0, 2)),
|
||||
None,
|
||||
Some(IntervalDayTime::new(0, 3)),
|
||||
]);
|
||||
let array: VectorRef = Arc::new(array);
|
||||
|
||||
let values = values(&[array]).unwrap();
|
||||
@@ -276,7 +283,12 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn test_convert_arrow_array_interval_month_day_nano() {
|
||||
let array = IntervalMonthDayNanoVector::from(vec![Some(1), Some(2), None, Some(3)]);
|
||||
let array = IntervalMonthDayNanoVector::from(vec![
|
||||
Some(IntervalMonthDayNano::new(0, 0, 1)),
|
||||
Some(IntervalMonthDayNano::new(0, 0, 2)),
|
||||
None,
|
||||
Some(IntervalMonthDayNano::new(0, 0, 3)),
|
||||
]);
|
||||
let array: VectorRef = Arc::new(array);
|
||||
|
||||
let values = values(&[array]).unwrap();
|
||||
|
||||
@@ -138,7 +138,7 @@ fn build_struct(
|
||||
datafusion_expr::create_udf(
|
||||
Self::name(),
|
||||
Self::input_type(),
|
||||
Arc::new(Self::return_type()),
|
||||
Self::return_type(),
|
||||
Volatility::Immutable,
|
||||
Arc::new(Self::calc) as _,
|
||||
)
|
||||
|
||||
@@ -47,7 +47,7 @@ pub(crate) fn build_template(create_table_expr: &CreateTableExpr) -> Result<Crea
|
||||
default_constraint: c.default_constraint.clone(),
|
||||
semantic_type: semantic_type as i32,
|
||||
comment: String::new(),
|
||||
datatype_extension: c.datatype_extension.clone(),
|
||||
datatype_extension: c.datatype_extension,
|
||||
options: c.options.clone(),
|
||||
}),
|
||||
column_id: i as u32,
|
||||
|
||||
@@ -378,6 +378,12 @@ impl FlowMetadataManager {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for FlowMetadataManager {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("FlowMetadataManager").finish()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::assert_matches::assert_matches;
|
||||
|
||||
@@ -290,7 +290,8 @@ mod tests {
|
||||
num_per_range: u32,
|
||||
max_bytes: u32,
|
||||
) {
|
||||
let num_cases = rand::thread_rng().gen_range(1..=26);
|
||||
let num_cases = rand::thread_rng().gen_range(1..=8);
|
||||
common_telemetry::info!("num_cases: {}", num_cases);
|
||||
let mut cases = Vec::with_capacity(num_cases);
|
||||
for i in 0..num_cases {
|
||||
let size = rand::thread_rng().gen_range(size_limit..=max_bytes);
|
||||
@@ -324,6 +325,7 @@ mod tests {
|
||||
|
||||
// Puts the values
|
||||
for TestCase { key, value, .. } in &cases {
|
||||
common_telemetry::info!("put key: {}, size: {}", key, value.len());
|
||||
store.put(key, value.clone()).await.unwrap();
|
||||
}
|
||||
|
||||
@@ -332,6 +334,7 @@ mod tests {
|
||||
let data = walk_top_down(prefix).await;
|
||||
assert_eq!(data.len(), 1);
|
||||
let (keyset, got) = data.into_iter().next().unwrap();
|
||||
common_telemetry::info!("get key: {}", keyset.key());
|
||||
let num_expected_keys = value.len().div_ceil(size_limit as usize);
|
||||
assert_eq!(&got, value);
|
||||
assert_eq!(keyset.key(), key);
|
||||
@@ -364,6 +367,7 @@ mod tests {
|
||||
let prefix = "test_etcd_store_split_value/";
|
||||
let endpoints = env::var("GT_ETCD_ENDPOINTS").unwrap_or_default();
|
||||
let kv_backend: KvBackendRef = if endpoints.is_empty() {
|
||||
common_telemetry::info!("Using MemoryKvBackend");
|
||||
Arc::new(MemoryKvBackend::new())
|
||||
} else {
|
||||
let endpoints = endpoints
|
||||
|
||||
@@ -237,6 +237,15 @@ pub enum Error {
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
},
|
||||
|
||||
#[snafu(display("Failed to register UDF: {}", name))]
|
||||
RegisterUdf {
|
||||
name: String,
|
||||
#[snafu(source)]
|
||||
error: DataFusionError,
|
||||
#[snafu(implicit)]
|
||||
location: Location,
|
||||
},
|
||||
}
|
||||
|
||||
pub type Result<T> = std::result::Result<T, Error>;
|
||||
@@ -268,7 +277,8 @@ impl ErrorExt for Error {
|
||||
|
||||
Error::MissingTableMutationHandler { .. }
|
||||
| Error::MissingProcedureServiceHandler { .. }
|
||||
| Error::MissingFlowServiceHandler { .. } => StatusCode::Unexpected,
|
||||
| Error::MissingFlowServiceHandler { .. }
|
||||
| Error::RegisterUdf { .. } => StatusCode::Unexpected,
|
||||
|
||||
Error::UnsupportedInputDataType { .. }
|
||||
| Error::TypeCast { .. }
|
||||
|
||||
@@ -214,8 +214,7 @@ mod tests {
|
||||
];
|
||||
|
||||
// call the function
|
||||
let result = (df_udf.fun())(&args).unwrap();
|
||||
|
||||
let result = df_udf.invoke_batch(&args, 4).unwrap();
|
||||
match result {
|
||||
DfColumnarValue::Array(arr) => {
|
||||
let arr = arr.as_any().downcast_ref::<BooleanArray>().unwrap();
|
||||
@@ -308,7 +307,7 @@ mod tests {
|
||||
Projection: person.id AS a, person.name AS b
|
||||
Filter: person.id > Int32(500)
|
||||
TableScan: person"#,
|
||||
format!("\n{:?}", new_plan)
|
||||
format!("\n{}", new_plan)
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -39,6 +39,10 @@ pub enum TypeSignature {
|
||||
Any(usize),
|
||||
/// One of a list of signatures
|
||||
OneOf(Vec<TypeSignature>),
|
||||
/// Zero argument
|
||||
/// This is the new signature for functions with zero arguments
|
||||
/// TODO(discord9): make all other usize nonzero usize
|
||||
NullAry,
|
||||
}
|
||||
|
||||
///The Signature of a function defines its supported input types as well as its volatility.
|
||||
@@ -112,6 +116,13 @@ impl Signature {
|
||||
volatility,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn nullary(volatility: Volatility) -> Self {
|
||||
Signature {
|
||||
type_signature: TypeSignature::NullAry,
|
||||
volatility,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Conversations between datafusion signature and our signature
|
||||
@@ -122,16 +133,25 @@ impl From<TypeSignature> for DfTypeSignature {
|
||||
DfTypeSignature::Variadic(concrete_types_to_arrow_types(types))
|
||||
}
|
||||
TypeSignature::Uniform(n, types) => {
|
||||
if n == 0 {
|
||||
return DfTypeSignature::NullAry;
|
||||
}
|
||||
DfTypeSignature::Uniform(n, concrete_types_to_arrow_types(types))
|
||||
}
|
||||
TypeSignature::Exact(types) => {
|
||||
DfTypeSignature::Exact(concrete_types_to_arrow_types(types))
|
||||
}
|
||||
TypeSignature::Any(n) => DfTypeSignature::Any(n),
|
||||
TypeSignature::Any(n) => {
|
||||
if n == 0 {
|
||||
return DfTypeSignature::NullAry;
|
||||
}
|
||||
DfTypeSignature::Any(n)
|
||||
}
|
||||
TypeSignature::OneOf(ts) => {
|
||||
DfTypeSignature::OneOf(ts.into_iter().map(Into::into).collect())
|
||||
}
|
||||
TypeSignature::VariadicAny => DfTypeSignature::VariadicAny,
|
||||
TypeSignature::NullAry => DfTypeSignature::NullAry,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -119,6 +119,10 @@ impl ExecutionPlan for StreamScanAdapter {
|
||||
.ok_or_else(|| DataFusionError::Execution("Stream already exhausted".to_string()))?;
|
||||
Ok(Box::pin(DfRecordBatchStreamAdapter::new(stream)))
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
"StreamScanAdapter"
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
@@ -17,6 +17,7 @@ use std::slice;
|
||||
use std::sync::Arc;
|
||||
|
||||
use datafusion::arrow::util::pretty::pretty_format_batches;
|
||||
use datatypes::arrow::array::RecordBatchOptions;
|
||||
use datatypes::prelude::DataType;
|
||||
use datatypes::schema::SchemaRef;
|
||||
use datatypes::value::Value;
|
||||
@@ -73,6 +74,21 @@ impl RecordBatch {
|
||||
}
|
||||
}
|
||||
|
||||
/// Create an empty [`RecordBatch`] from `schema` with `num_rows`.
|
||||
pub fn new_with_count(schema: SchemaRef, num_rows: usize) -> Result<Self> {
|
||||
let df_record_batch = DfRecordBatch::try_new_with_options(
|
||||
schema.arrow_schema().clone(),
|
||||
vec![],
|
||||
&RecordBatchOptions::new().with_row_count(Some(num_rows)),
|
||||
)
|
||||
.context(error::NewDfRecordBatchSnafu)?;
|
||||
Ok(RecordBatch {
|
||||
schema,
|
||||
columns: vec![],
|
||||
df_record_batch,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn try_project(&self, indices: &[usize]) -> Result<Self> {
|
||||
let schema = Arc::new(self.schema.try_project(indices).context(DataTypesSnafu)?);
|
||||
let mut columns = Vec::with_capacity(indices.len());
|
||||
|
||||
@@ -142,16 +142,25 @@ impl Builder {
|
||||
|
||||
impl BuilderBuild<DefaultRuntime> for Builder {
|
||||
fn build(&mut self) -> Result<DefaultRuntime> {
|
||||
let runtime = self
|
||||
let builder = self
|
||||
.builder
|
||||
.enable_all()
|
||||
.thread_name(self.thread_name.clone())
|
||||
.on_thread_start(on_thread_start(self.thread_name.clone()))
|
||||
.on_thread_stop(on_thread_stop(self.thread_name.clone()))
|
||||
.on_thread_park(on_thread_park(self.thread_name.clone()))
|
||||
.on_thread_unpark(on_thread_unpark(self.thread_name.clone()))
|
||||
.build()
|
||||
.context(BuildRuntimeSnafu)?;
|
||||
.on_thread_unpark(on_thread_unpark(self.thread_name.clone()));
|
||||
let runtime = if cfg!(debug_assertions) {
|
||||
// Set the stack size to 8MB for the thread so it wouldn't overflow on large stack usage in debug mode
|
||||
// This is necessary to avoid stack overflow while running sqlness.
|
||||
// https://github.com/rust-lang/rust/issues/34283
|
||||
builder
|
||||
.thread_stack_size(8 * 1024 * 1024)
|
||||
.build()
|
||||
.context(BuildRuntimeSnafu)?
|
||||
} else {
|
||||
builder.build().context(BuildRuntimeSnafu)?
|
||||
};
|
||||
|
||||
let name = self.runtime_name.clone();
|
||||
let handle = runtime.handle().clone();
|
||||
|
||||
@@ -16,10 +16,10 @@ use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use bytes::{Buf, Bytes, BytesMut};
|
||||
use datafusion::catalog::CatalogProviderList;
|
||||
use datafusion::execution::context::SessionState;
|
||||
use datafusion::execution::runtime_env::RuntimeEnv;
|
||||
use datafusion::prelude::{SessionConfig, SessionContext};
|
||||
use datafusion::execution::SessionStateBuilder;
|
||||
use datafusion::prelude::SessionConfig;
|
||||
use datafusion_expr::LogicalPlan;
|
||||
use datafusion_substrait::logical_plan::consumer::from_substrait_plan;
|
||||
use datafusion_substrait::logical_plan::producer::to_substrait_plan;
|
||||
@@ -41,13 +41,10 @@ impl SubstraitPlan for DFLogicalSubstraitConvertor {
|
||||
async fn decode<B: Buf + Send>(
|
||||
&self,
|
||||
message: B,
|
||||
catalog_list: Arc<dyn CatalogProviderList>,
|
||||
state: SessionState,
|
||||
) -> Result<Self::Plan, Self::Error> {
|
||||
let mut context = SessionContext::new_with_state(state);
|
||||
context.register_catalog_list(catalog_list);
|
||||
let plan = Plan::decode(message).context(DecodeRelSnafu)?;
|
||||
let df_plan = from_substrait_plan(&context, &plan)
|
||||
let df_plan = from_substrait_plan(&state, &plan)
|
||||
.await
|
||||
.context(DecodeDfPlanSnafu)?;
|
||||
Ok(df_plan)
|
||||
@@ -72,11 +69,12 @@ impl DFLogicalSubstraitConvertor {
|
||||
plan: &LogicalPlan,
|
||||
serializer: impl SerializerRegistry + 'static,
|
||||
) -> Result<Box<Plan>, Error> {
|
||||
let session_state =
|
||||
SessionState::new_with_config_rt(SessionConfig::new(), Arc::new(RuntimeEnv::default()))
|
||||
.with_serializer_registry(Arc::new(serializer));
|
||||
let context = SessionContext::new_with_state(session_state);
|
||||
|
||||
to_substrait_plan(plan, &context).context(EncodeDfPlanSnafu)
|
||||
let state = SessionStateBuilder::new()
|
||||
.with_config(SessionConfig::new())
|
||||
.with_runtime_env(Arc::new(RuntimeEnv::default()))
|
||||
.with_default_features()
|
||||
.with_serializer_registry(Arc::new(serializer))
|
||||
.build();
|
||||
to_substrait_plan(plan, &state).context(EncodeDfPlanSnafu)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@ use promql::extension_plan::{
|
||||
EmptyMetric, InstantManipulate, RangeManipulate, ScalarCalculate, SeriesDivide, SeriesNormalize,
|
||||
};
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct ExtensionSerializer;
|
||||
|
||||
impl SerializerRegistry for ExtensionSerializer {
|
||||
|
||||
@@ -17,11 +17,9 @@
|
||||
mod df_substrait;
|
||||
pub mod error;
|
||||
pub mod extension_serializer;
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use bytes::{Buf, Bytes};
|
||||
use datafusion::catalog::CatalogProviderList;
|
||||
use datafusion::execution::context::SessionState;
|
||||
pub use datafusion::execution::registry::SerializerRegistry;
|
||||
/// Re-export the Substrait module of datafusion,
|
||||
@@ -40,7 +38,6 @@ pub trait SubstraitPlan {
|
||||
async fn decode<B: Buf + Send>(
|
||||
&self,
|
||||
message: B,
|
||||
catalog_list: Arc<dyn CatalogProviderList>,
|
||||
state: SessionState,
|
||||
) -> Result<Self::Plan, Self::Error>;
|
||||
|
||||
|
||||
@@ -22,6 +22,6 @@ static PORTS: OnceCell<AtomicUsize> = OnceCell::new();
|
||||
/// Return a unique port(in runtime) for test
|
||||
pub fn get_port() -> usize {
|
||||
PORTS
|
||||
.get_or_init(|| AtomicUsize::new(rand::thread_rng().gen_range(3000..3800)))
|
||||
.get_or_init(|| AtomicUsize::new(rand::thread_rng().gen_range(13000..13800)))
|
||||
.fetch_add(1, Ordering::Relaxed)
|
||||
}
|
||||
|
||||
@@ -184,6 +184,24 @@ impl From<IntervalDayTime> for serde_json::Value {
|
||||
}
|
||||
}
|
||||
|
||||
impl From<arrow::datatypes::IntervalDayTime> for IntervalDayTime {
|
||||
fn from(value: arrow::datatypes::IntervalDayTime) -> Self {
|
||||
Self {
|
||||
days: value.days,
|
||||
milliseconds: value.milliseconds,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<IntervalDayTime> for arrow::datatypes::IntervalDayTime {
|
||||
fn from(value: IntervalDayTime) -> Self {
|
||||
Self {
|
||||
days: value.days,
|
||||
milliseconds: value.milliseconds,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Millisecond convert to other time unit
|
||||
pub const MS_PER_SEC: i64 = 1_000;
|
||||
pub const MS_PER_MINUTE: i64 = 60 * MS_PER_SEC;
|
||||
@@ -283,6 +301,26 @@ impl From<IntervalMonthDayNano> for serde_json::Value {
|
||||
}
|
||||
}
|
||||
|
||||
impl From<arrow::datatypes::IntervalMonthDayNano> for IntervalMonthDayNano {
|
||||
fn from(value: arrow::datatypes::IntervalMonthDayNano) -> Self {
|
||||
Self {
|
||||
months: value.months,
|
||||
days: value.days,
|
||||
nanoseconds: value.nanoseconds,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<IntervalMonthDayNano> for arrow::datatypes::IntervalMonthDayNano {
|
||||
fn from(value: IntervalMonthDayNano) -> Self {
|
||||
Self {
|
||||
months: value.months,
|
||||
days: value.days,
|
||||
nanoseconds: value.nanoseconds,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Nanosecond convert to other time unit
|
||||
pub const NS_PER_SEC: i64 = 1_000_000_000;
|
||||
pub const NS_PER_MINUTE: i64 = 60 * NS_PER_SEC;
|
||||
|
||||
Reference in New Issue
Block a user