mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-08-18 12:08:22 +00:00
feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: upgrading network deps Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adapt new df's trait requirements Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi <jiachun_feng@proton.me> * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: luofucong <luofc@foxmail.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: jeremyhi <jiachun_feng@proton.me>
This commit is contained in:
Yingwen
GitHub
@@ -64,25 +64,21 @@ macro_rules! impl_min_max_values {
|
||||
.iter()
|
||||
.map(|meta| {
|
||||
let stats = meta.column(column_index).statistics()?;
|
||||
if !stats.has_min_max_set() {
|
||||
return None;
|
||||
}
|
||||
match stats {
|
||||
ParquetStats::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.$min_max()))),
|
||||
ParquetStats::Int32(s) => Some(ScalarValue::Int32(Some(*s.$min_max()))),
|
||||
ParquetStats::Int64(s) => Some(ScalarValue::Int64(Some(*s.$min_max()))),
|
||||
paste! {
|
||||
match stats {
|
||||
ParquetStats::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.[<$min_max _opt>]()?))),
|
||||
ParquetStats::Int32(s) => Some(ScalarValue::Int32(Some(*s.[<$min_max _opt>]()?))),
|
||||
ParquetStats::Int64(s) => Some(ScalarValue::Int64(Some(*s.[<$min_max _opt>]()?))),
|
||||
|
||||
ParquetStats::Int96(_) => None,
|
||||
ParquetStats::Float(s) => Some(ScalarValue::Float32(Some(*s.$min_max()))),
|
||||
ParquetStats::Double(s) => Some(ScalarValue::Float64(Some(*s.$min_max()))),
|
||||
ParquetStats::ByteArray(s) => {
|
||||
paste! {
|
||||
let s = String::from_utf8(s.[<$min_max _bytes>]().to_owned()).ok();
|
||||
ParquetStats::Int96(_) => None,
|
||||
ParquetStats::Float(s) => Some(ScalarValue::Float32(Some(*s.[<$min_max _opt>]()?))),
|
||||
ParquetStats::Double(s) => Some(ScalarValue::Float64(Some(*s.[<$min_max _opt>]()?))),
|
||||
ParquetStats::ByteArray(s) => {
|
||||
let s = String::from_utf8(s.[<$min_max _bytes_opt>]()?.to_owned()).ok();
|
||||
Some(ScalarValue::Utf8(s))
|
||||
}
|
||||
Some(ScalarValue::Utf8(s))
|
||||
ParquetStats::FixedLenByteArray(_) => None,
|
||||
}
|
||||
|
||||
ParquetStats::FixedLenByteArray(_) => None,
|
||||
}
|
||||
})
|
||||
.map(|maybe_scalar| maybe_scalar.unwrap_or_else(|| null_scalar.clone()))
|
||||
@@ -111,7 +107,7 @@ impl PruningStatistics for RowGroupPruningStatistics<'_> {
|
||||
for m in self.meta_data {
|
||||
let col = m.column(idx);
|
||||
let stat = col.statistics()?;
|
||||
let bs = stat.null_count();
|
||||
let bs = stat.null_count_opt()?;
|
||||
values.push(Some(bs));
|
||||
}
|
||||
Some(Arc::new(UInt64Array::from(values)))
|
||||
|
||||
@@ -18,9 +18,9 @@ use std::sync::{Arc, Mutex};
|
||||
use common_query::stream::StreamScanAdapter;
|
||||
use common_recordbatch::OrderOption;
|
||||
use datafusion::arrow::datatypes::SchemaRef as DfSchemaRef;
|
||||
use datafusion::catalog::Session;
|
||||
use datafusion::datasource::{TableProvider, TableType as DfTableType};
|
||||
use datafusion::error::Result as DfResult;
|
||||
use datafusion::execution::context::SessionState;
|
||||
use datafusion::physical_plan::ExecutionPlan;
|
||||
use datafusion_expr::expr::Expr;
|
||||
use datafusion_expr::TableProviderFilterPushDown as DfTableProviderFilterPushDown;
|
||||
@@ -58,6 +58,14 @@ impl DfTableProviderAdapter {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for DfTableProviderAdapter {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("DfTableProviderAdapter")
|
||||
.field("table", &self.table.table_info.full_table_name())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TableProvider for DfTableProviderAdapter {
|
||||
fn as_any(&self) -> &dyn Any {
|
||||
@@ -82,7 +90,7 @@ impl TableProvider for DfTableProviderAdapter {
|
||||
|
||||
async fn scan(
|
||||
&self,
|
||||
_ctx: &SessionState,
|
||||
_state: &dyn Session,
|
||||
projection: Option<&Vec<usize>>,
|
||||
filters: &[Expr],
|
||||
limit: Option<usize>,
|
||||
|
||||
@@ -105,10 +105,16 @@ impl NumbersDataSource {
|
||||
|
||||
impl DataSource for NumbersDataSource {
|
||||
fn get_stream(&self, request: ScanRequest) -> Result<SendableRecordBatchStream, BoxedError> {
|
||||
let projected_schema = match &request.projection {
|
||||
Some(projection) => Arc::new(self.schema.try_project(projection).unwrap()),
|
||||
None => self.schema.clone(),
|
||||
};
|
||||
Ok(Box::pin(NumbersStream {
|
||||
limit: request.limit.unwrap_or(100) as u32,
|
||||
schema: self.schema.clone(),
|
||||
already_run: false,
|
||||
projection: request.projection,
|
||||
projected_schema,
|
||||
}))
|
||||
}
|
||||
}
|
||||
@@ -118,11 +124,13 @@ struct NumbersStream {
|
||||
limit: u32,
|
||||
schema: SchemaRef,
|
||||
already_run: bool,
|
||||
projection: Option<Vec<usize>>,
|
||||
projected_schema: SchemaRef,
|
||||
}
|
||||
|
||||
impl RecordBatchStream for NumbersStream {
|
||||
fn schema(&self) -> SchemaRef {
|
||||
self.schema.clone()
|
||||
self.projected_schema.clone()
|
||||
}
|
||||
|
||||
fn output_ordering(&self) -> Option<&[OrderOption]> {
|
||||
@@ -143,14 +151,17 @@ impl Stream for NumbersStream {
|
||||
}
|
||||
self.already_run = true;
|
||||
let numbers: Vec<u32> = (0..self.limit).collect();
|
||||
let batch = DfRecordBatch::try_new(
|
||||
let mut batch = DfRecordBatch::try_new(
|
||||
self.schema.arrow_schema().clone(),
|
||||
vec![Arc::new(UInt32Array::from(numbers))],
|
||||
)
|
||||
.unwrap();
|
||||
if let Some(projection) = &self.projection {
|
||||
batch = batch.project(projection).unwrap();
|
||||
}
|
||||
|
||||
Poll::Ready(Some(RecordBatch::try_from_df_record_batch(
|
||||
self.schema.clone(),
|
||||
self.projected_schema.clone(),
|
||||
batch,
|
||||
)))
|
||||
}
|
||||
|
||||
@@ -247,6 +247,10 @@ impl ExecutionPlan for RegionScanExec {
|
||||
};
|
||||
Ok(statistics)
|
||||
}
|
||||
|
||||
fn name(&self) -> &str {
|
||||
"RegionScanExec"
|
||||
}
|
||||
}
|
||||
|
||||
impl DisplayAs for RegionScanExec {
|
||||
|
||||
Reference in New Issue
Block a user