From 35b635f6399dc0ee33bc61d7b22d6dd76b3b5626 Mon Sep 17 00:00:00 2001 From: Yingwen Date: Thu, 23 Jan 2025 14:15:40 +0800 Subject: [PATCH] feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417) * change dep Signed-off-by: Ruihang Xia * feat: adapt to arrow's interval array * chore: fix compile errors in datatypes crate * chore: fix api crate compiler errors * chore: fix compiler errors in common-grpc * chore: fix common-datasource errors * chore: fix deprecated code in common-datasource * fix promql and physical plan related Signed-off-by: Ruihang Xia * wip: upgrading network deps Signed-off-by: Ruihang Xia * block on updating `sqlparser` * upgrade sqlparser Signed-off-by: Ruihang Xia * adapt new df's trait requirements Signed-off-by: Ruihang Xia * chore: fix compiler errors in mito2 * chore: fix common-function crate errors * chore: fix catalog errors * change import path Signed-off-by: Ruihang Xia * chore: fix some errors in query crate * chore: fix some errors in query crate * aggr expr and some other tiny fixes Signed-off-by: Ruihang Xia * chore: fix expr related errors in query crate * chore: fix query serializer and admin command * chore: fix grpc services * feat: axum serve * chore: fix http server * remove handle_error handler * refactor timeout layer * serve axum * chore: fix flow aggr functions * chore: fix flow * feat: fix errors in meta-srv * boxed() * use TokioIo * feat!: Remove script crate and python feature (#5321) * feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning * chore: fix servers tests compiler errors * feat: fix tests-integration errors * chore: fix unused * test: fix catalog test * chore: fix compiler errors for crates using common-meta testing feature is enabled when check with --workspace * test: use display for logical plan test * test: implement rewrite for ScanHintRule * fix: http server build panic * test: fix mito test * fix: sql parser type alias error * test: fix TestClient not listen * test: some flow tests * test(flow): more fix * fix: test_otlp_logs * test: fix promql test that using deprecated method fun() * fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64 * test: fix infer schema test case * test: fix tests related to plan display * chore: fix last flow test * test: fix function format related assertion * test: use larger port range for tests * fix: test_otlp_traces * fix: test_otlp_metrics * fix range query and dist plan Signed-off-by: Ruihang Xia * fix: flow handle distinct use deprecated field * fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs * test: fix deserialize test * test: reduce split key case num * tests: lower case aggr func name * test: fix some sqlness tests * tests: more sqlness fix * tests: fixed sqlness test * commit non-bug changes Signed-off-by: Ruihang Xia * fix: make our udf correct * fix: implement empty methods of ContextProvider for DfContextProviderAdapter * test: update sqlness test result * chore: remove unused * fix: provide alias name for AggregateExprBuilder in range plan * test: update range query result * fix: implement missing ContextProvider methods for DfContextProviderAdapter * test: update timestamps, cte result * fix: supports empty projection in mito * test: update comment for cte test * fix: support projection for numbers * test: update test cases after projection fix * fix: fix range select first_value/last_value * fix: handle CAST and time index conflict * fix: handle order by correctly in range first_value/last_value * test: update sqlness result * test: update view test result * test: update decimal test wait for https://github.com/apache/datafusion/pull/14126 to fix this * feat: remove redundant physical optimization todo(ruihang): Check if we can remove this. * test: update sqlness test result * chore: range select default sort use nulls_first = false * test: update filter push down test result * test: comment deciaml test to avoid different panic message * test: update some distributed test result * test: update test for distributed count and filter push down * test: update subqueries test * fix: SessionState may overwrite our UDFs * chore: fix compiler errors after merging main * fix: fix elasticsearch and dashboard router panic * chore: fix common-functions tests * chore: update sqlness result * test: fix id keyword and update sqlness result * test: fix flow_null test * fix: enlarge thread size in debug mode to avoid overflow * chore: fix warnings in common-function * chore: fix warning in flow * chore: fix warnings in query crate * chore: remove unused warnings * chore: fix deprecated warnings for parquet * chore: fix deprecated warning in servers crate * style: fix clippy * test: enlarge mito cache tttl test ttl time * chore: fix typo * style: fmt toml * refactor: reimplement PartialOrd for RangeSelect * chore: remove script crate files introduced by merge * fix: return error if sql option is not kv * chore: do not use ..default::default() * chore: per review * chore: update error message in BuildAdminFunctionArgsSnafu Co-authored-by: jeremyhi * refactor: typed precision * update sqlness view case Signed-off-by: Ruihang Xia * chore: flow per review * chore: add example in comment * chore: warn if parquet stats of timestamp is not INT64 * style: add a newline before derive to make the comment more clear * test: update sqlness result * fix: flow from substrait * chore: change update_range_context log to debug level * chore: move axum-extra axum-macros to workspace --------- Signed-off-by: Ruihang Xia Co-authored-by: Ruihang Xia Co-authored-by: luofucong Co-authored-by: discord9 Co-authored-by: shuiyisong Co-authored-by: jeremyhi --- Cargo.lock | 1554 +++++++++-------- Cargo.toml | 58 +- src/api/src/error.rs | 2 +- src/api/src/helper.rs | 28 +- src/api/src/v1/column_def.rs | 6 +- src/catalog/src/lib.rs | 1 + .../information_schema/cluster_info.rs | 1 + .../information_schema/columns.rs | 1 + .../system_schema/information_schema/flows.rs | 2 +- .../information_schema/key_column_usage.rs | 1 + .../information_schema/partitions.rs | 1 + .../information_schema/procedure_info.rs | 2 +- .../information_schema/region_peers.rs | 2 +- .../information_schema/region_statistics.rs | 2 +- .../information_schema/runtime_metrics.rs | 1 + .../information_schema/schemata.rs | 1 + .../information_schema/table_constraints.rs | 1 + .../information_schema/tables.rs | 1 + .../system_schema/information_schema/views.rs | 1 + src/catalog/src/system_schema/memory_table.rs | 1 + .../src/system_schema/pg_catalog/pg_class.rs | 10 + .../system_schema/pg_catalog/pg_database.rs | 9 + .../system_schema/pg_catalog/pg_namespace.rs | 10 + src/catalog/src/table_source.rs | 2 +- src/catalog/src/table_source/dummy_catalog.rs | 27 +- src/common/datasource/Cargo.toml | 2 +- src/common/datasource/src/file_format.rs | 3 +- src/common/datasource/src/file_format/csv.rs | 15 +- src/common/datasource/src/file_format/json.rs | 6 +- .../function/src/scalars/date/date_add.rs | 8 +- .../function/src/scalars/date/date_sub.rs | 8 +- src/common/function/src/scalars/matches.rs | 8 +- src/common/function/src/system/build.rs | 11 +- src/common/function/src/system/database.rs | 10 +- src/common/function/src/system/timezone.rs | 10 +- src/common/function/src/system/version.rs | 2 +- src/common/grpc-expr/src/error.rs | 4 +- src/common/grpc-expr/src/insert.rs | 3 +- src/common/grpc-expr/src/util.rs | 4 +- src/common/grpc/Cargo.toml | 4 +- src/common/grpc/src/channel_manager.rs | 8 +- src/common/grpc/src/select.rs | 18 +- src/common/macro/src/range_fn.rs | 2 +- .../meta/src/ddl/create_table_template.rs | 2 +- src/common/meta/src/key/flow.rs | 6 + src/common/meta/src/state_store.rs | 6 +- src/common/query/src/error.rs | 12 +- src/common/query/src/logical_plan.rs | 5 +- src/common/query/src/signature.rs | 22 +- src/common/query/src/stream.rs | 4 + src/common/recordbatch/src/recordbatch.rs | 16 + src/common/runtime/src/runtime.rs | 17 +- src/common/substrait/src/df_substrait.rs | 22 +- .../substrait/src/extension_serializer.rs | 1 + src/common/substrait/src/lib.rs | 3 - src/common/test-util/src/ports.rs | 2 +- src/common/time/src/interval.rs | 38 + src/datatypes/src/interval.rs | 18 +- src/datatypes/src/schema/column_schema.rs | 54 +- src/datatypes/src/types/interval_type.rs | 8 +- src/datatypes/src/value.rs | 19 +- src/datatypes/src/vectors/eq.rs | 32 +- src/datatypes/src/vectors/helper.rs | 20 +- src/datatypes/src/vectors/primitive.rs | 14 +- src/flow/Cargo.toml | 1 + src/flow/src/adapter/util.rs | 10 +- src/flow/src/df_optimizer.rs | 24 +- src/flow/src/expr/func.rs | 14 +- src/flow/src/expr/relation/func.rs | 33 +- src/flow/src/plan.rs | 2 + src/flow/src/transform.rs | 15 +- src/flow/src/transform/aggr.rs | 154 +- src/flow/src/transform/expr.rs | 68 +- src/flow/src/transform/literal.rs | 316 ++-- src/flow/src/transform/plan.rs | 4 +- src/index/src/bloom_filter/applier.rs | 2 +- src/meta-srv/Cargo.toml | 4 +- src/meta-srv/src/mocks.rs | 3 +- src/meta-srv/src/procedure/utils.rs | 3 +- src/meta-srv/src/service/admin.rs | 10 +- src/mito2/src/cache/cache_size.rs | 3 +- src/mito2/src/cache/file_cache.rs | 4 +- src/mito2/src/engine/basic_test.rs | 6 +- .../src/memtable/bulk/row_group_reader.rs | 20 +- src/mito2/src/read/projection.rs | 65 +- src/mito2/src/request.rs | 2 +- src/mito2/src/sst/parquet/file_range.rs | 26 +- src/mito2/src/sst/parquet/format.rs | 85 +- src/mito2/src/sst/parquet/metadata.rs | 25 +- src/mito2/src/sst/parquet/row_group.rs | 36 +- src/operator/src/req_convert/common.rs | 4 +- src/operator/src/request.rs | 2 +- src/operator/src/statement/admin.rs | 12 +- src/operator/src/statement/ddl.rs | 2 +- src/partition/src/manager.rs | 2 +- src/promql/src/extension_plan/empty_metric.rs | 23 + .../src/extension_plan/histogram_fold.rs | 35 +- .../src/extension_plan/instant_manipulate.rs | 8 +- src/promql/src/extension_plan/normalize.rs | 6 +- .../src/extension_plan/range_manipulate.rs | 35 + .../src/extension_plan/scalar_calculate.rs | 35 + .../src/extension_plan/series_divide.rs | 12 +- .../src/extension_plan/union_distinct_on.rs | 23 + src/promql/src/functions/extrapolate_rate.rs | 2 +- src/promql/src/functions/holt_winters.rs | 2 +- src/promql/src/functions/idelta.rs | 2 +- src/promql/src/functions/predict_linear.rs | 2 +- src/promql/src/functions/quantile.rs | 2 +- src/promql/src/functions/test_util.rs | 16 +- src/query/src/datafusion.rs | 64 +- src/query/src/datafusion/planner.rs | 87 +- src/query/src/dist_plan/analyzer.rs | 28 +- src/query/src/dist_plan/commutativity.rs | 10 +- src/query/src/dist_plan/merge_scan.rs | 6 +- src/query/src/dist_plan/merge_sort.rs | 17 +- src/query/src/dummy_catalog.rs | 28 +- src/query/src/optimizer/count_wildcard.rs | 36 +- src/query/src/optimizer/parallelize_scan.rs | 1 + src/query/src/optimizer/remove_duplicate.rs | 1 + src/query/src/optimizer/scan_hint.rs | 75 +- .../src/optimizer/string_normalization.rs | 13 +- src/query/src/optimizer/test_util.rs | 3 +- src/query/src/optimizer/type_conversion.rs | 29 +- src/query/src/optimizer/windowed_sort.rs | 4 +- src/query/src/parser.rs | 33 +- src/query/src/part_sort.rs | 52 +- src/query/src/plan.rs | 47 +- src/query/src/planner.rs | 8 + src/query/src/promql/planner.rs | 77 +- .../src/query_engine/default_serializer.rs | 68 +- src/query/src/query_engine/state.rs | 41 +- src/query/src/range_select/plan.rs | 375 ++-- src/query/src/range_select/plan_rewrite.rs | 85 +- src/query/src/sql.rs | 10 +- src/query/src/test_util.rs | 4 + src/query/src/window_sort.rs | 49 +- src/servers/Cargo.toml | 15 +- src/servers/src/elasticsearch.rs | 5 +- src/servers/src/error.rs | 11 +- src/servers/src/grpc.rs | 8 +- src/servers/src/grpc/authorize.rs | 23 +- src/servers/src/grpc/builder.rs | 2 +- src/servers/src/http.rs | 122 +- src/servers/src/http/authorize.rs | 15 +- src/servers/src/http/dashboard.rs | 6 +- src/servers/src/http/event.rs | 23 +- src/servers/src/http/extractor.rs | 16 +- src/servers/src/http/hints.rs | 3 +- src/servers/src/http/loki.rs | 5 +- src/servers/src/http/opentsdb.rs | 19 +- src/servers/src/http/prom_store.rs | 32 +- .../src/http/result/greptime_manage_resp.rs | 14 +- .../src/http/result/greptime_result_v1.rs | 2 +- src/servers/src/http/test_helpers.rs | 59 +- src/servers/src/http/timeout.rs | 40 +- src/servers/src/metrics.rs | 12 +- src/servers/src/mysql/helper.rs | 4 +- src/servers/src/postgres/types.rs | 14 +- src/servers/tests/http/authorize.rs | 20 +- src/servers/tests/http/http_handler_test.rs | 58 +- src/servers/tests/http/influxdb_test.rs | 12 +- src/servers/tests/http/opentsdb_test.rs | 8 +- src/servers/tests/http/prom_store_test.rs | 2 +- src/sql/src/ast.rs | 5 +- src/sql/src/error.rs | 4 +- src/sql/src/lib.rs | 1 + src/sql/src/parser.rs | 2 +- src/sql/src/parsers/admin_parser.rs | 14 +- src/sql/src/parsers/alter_parser.rs | 4 +- src/sql/src/parsers/create_parser.rs | 21 +- src/sql/src/parsers/explain_parser.rs | 13 +- src/sql/src/parsers/query_parser.rs | 4 +- src/sql/src/parsers/set_var_parser.rs | 7 +- src/sql/src/parsers/utils.rs | 7 +- src/sql/src/statements.rs | 4 +- src/sql/src/statements/create.rs | 4 +- src/sql/src/statements/insert.rs | 20 +- src/sql/src/statements/transform.rs | 11 +- .../statements/transform/expand_interval.rs | 3 + .../src/statements/transform/type_alias.rs | 109 +- src/sql/src/util.rs | 9 +- src/store-api/src/metadata.rs | 7 +- src/table/src/predicate/stats.rs | 30 +- src/table/src/table/adapter.rs | 12 +- src/table/src/table/numbers.rs | 17 +- src/table/src/table/scan.rs | 4 + tests-integration/Cargo.toml | 3 +- tests-integration/src/cluster.rs | 3 +- tests-integration/src/otlp.rs | 2 + tests-integration/src/tests/promql_test.rs | 18 +- tests-integration/tests/http.rs | 68 +- .../cases/distributed/explain/analyze.result | 13 +- tests/cases/distributed/explain/analyze.sql | 2 + .../explain/analyze_append_table_count.result | 15 +- .../explain/analyze_append_table_count.sql | 5 + .../distributed/explain/join_10_tables.result | 6 +- .../cases/distributed/explain/order_by.result | 4 +- .../distributed/explain/subqueries.result | 31 +- .../cases/distributed/explain/subqueries.sql | 4 + .../optimizer/filter_push_down.result | 12 +- .../optimizer/filter_push_down.sql | 2 + .../distributed/optimizer/order_by.result | 34 +- .../standalone/common/aggregate/count.result | 22 +- .../standalone/common/aggregate/count.sql | 8 +- .../common/aggregate/distinct.result | 4 +- .../common/aggregate/multi_regions.result | 25 +- .../standalone/common/aggregate/sum.result | 14 +- .../common/alter/change_col_type.result | 2 +- .../common/alter/change_col_type.sql | 2 +- tests/cases/standalone/common/basic.result | 6 +- .../common/copy/copy_from_fs_parquet.result | 6 +- .../common/create/create_if_not_exists.result | 4 +- .../common/create/create_with_options.result | 10 +- .../common/create/create_with_options.sql | 10 +- .../create/create_with_skipping_index.result | 2 +- .../create/create_with_skipping_index.sql | 2 +- tests/cases/standalone/common/cte/cte.result | 7 +- tests/cases/standalone/common/cte/cte.sql | 1 - .../common/flow/flow_auto_sink_table.result | 4 +- .../standalone/common/flow/flow_basic.result | 104 +- .../standalone/common/flow/flow_basic.sql | 16 +- .../common/flow/flow_call_df_func.result | 16 +- .../common/flow/flow_call_df_func.sql | 8 +- .../standalone/common/flow/flow_null.result | 60 +- .../standalone/common/flow/flow_null.sql | 12 +- .../standalone/common/function/date.result | 96 +- .../standalone/common/function/geo.result | 20 +- .../information_schema/region_peers.result | 2 +- .../region_statistics.result | 2 +- .../common/insert/big_insert.result | 2 +- .../standalone/common/order/limit.result | 28 +- .../common/order/limit_union.result | 2 +- .../common/order/nulls_first.result | 2 +- .../standalone/common/order/order_by.result | 23 +- .../common/order/order_by_exceptions.result | 2 +- .../common/order/windowed_sort.result | 12 +- .../standalone/common/parser/parser.result | 2 +- .../common/prepare/mysql_prepare.result | 2 +- .../standalone/common/promql/scalar.result | 2 +- .../common/promql/simple_histogram.result | 2 +- .../standalone/common/promql/time_fn.result | 154 +- tests/cases/standalone/common/range/by.result | 12 +- .../standalone/common/range/calculate.result | 16 +- .../standalone/common/range/error.result | 18 +- .../cases/standalone/common/range/fill.result | 30 +- .../standalone/common/range/interval.result | 18 +- .../cases/standalone/common/range/nest.result | 8 +- .../standalone/common/range/precisions.result | 2 +- .../common/range/special_aggr.result | 14 +- tests/cases/standalone/common/range/to.result | 46 +- .../common/select/flush_append_only.result | 2 +- .../common/select/prune_field.result | 8 +- .../standalone/common/select/prune_field.sql | 8 +- .../common/select/skipping_index.result | 4 +- .../common/select/skipping_index.sql | 2 +- .../standalone/common/show/show_create.result | 24 +- .../standalone/common/show/show_create.sql | 24 +- .../standalone/common/subquery/table.result | 4 +- .../common/subquery/test_neumann.result | 8 +- .../common/subquery/test_neumann.sql | 8 +- .../common/system/information_schema.result | 2 +- .../common/tql-explain-analyze/analyze.result | 20 +- .../common/tql-explain-analyze/explain.result | 20 +- .../common/tql/binary_operator.result | 2 +- .../common/tql/case_sensitive.result | 2 +- tests/cases/standalone/common/tql/join.result | 10 +- .../types/decimal/decimal_aggregates.result | 10 +- .../types/decimal/decimal_arithmetic.result | 6 +- .../types/decimal/decimal_arithmetic.sql | 3 +- .../common/types/decimal/decimal_cast.result | 8 +- .../common/types/float/nan_aggregate.result | 20 +- .../common/types/float/nan_aggregates.result | 16 +- .../common/types/float/nan_ordering.result | 8 +- .../common/types/float/nan_window.result | 16 +- .../common/types/interval/interval.result | 296 ++-- .../common/types/interval/interval.sql | 8 +- .../types/string/scan_big_varchar.result | 34 +- .../common/types/string/unicode.result | 2 +- .../common/types/string/unicode.sql | 2 +- .../common/types/timestamp/timestamp.result | 18 +- .../types/timestamp/timestamp_join.result | 2 +- .../timestamp/timestamp_precision.result | 18 +- .../types/timestamp/timestamp_precision.sql | 2 +- .../types/timestamp/timestamp_types.result | 8 +- .../standalone/common/view/columns.result | 18 +- .../cases/standalone/common/view/columns.sql | 8 - .../standalone/common/view/create.result | 4 +- .../cases/standalone/common/view/view.result | 37 +- tests/cases/standalone/common/view/view.sql | 24 +- .../optimizer/filter_push_down.result | 10 +- .../standalone/optimizer/filter_push_down.sql | 2 + .../standalone/optimizer/last_value.result | 1 - .../standalone/optimizer/order_by.result | 34 +- 293 files changed, 4067 insertions(+), 3225 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1fc3a19ee2..98ac14afd2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -195,10 +195,10 @@ dependencies = [ "datatypes", "greptime-proto", "paste", - "prost 0.12.6", + "prost 0.13.3", "serde_json", "snafu 0.8.5", - "tonic-build", + "tonic-build 0.11.0", ] [[package]] @@ -268,9 +268,9 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "219d05930b81663fd3b32e3bde8ce5bff3c4d23052a99f11a8fa50a3b47b2658" +checksum = "c91839b07e474b3995035fd8ac33ee54f9c9ccbbb1ea33d9909c71bffdf1259d" dependencies = [ "arrow-arith", "arrow-array", @@ -289,9 +289,9 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0272150200c07a86a390be651abdd320a2d12e84535f0837566ca87ecd8f95e0" +checksum = "855c57c4efd26722b044dcd3e348252560e3e0333087fb9f6479dc0bf744054f" dependencies = [ "arrow-array", "arrow-buffer", @@ -304,26 +304,26 @@ dependencies = [ [[package]] name = "arrow-array" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8010572cf8c745e242d1b632bd97bd6d4f40fefed5ed1290a8f433abaa686fea" +checksum = "bd03279cea46569acf9295f6224fbc370c5df184b4d2ecfe97ccb131d5615a7f" dependencies = [ "ahash 0.8.11", "arrow-buffer", "arrow-data", "arrow-schema", "chrono", - "chrono-tz 0.8.6", + "chrono-tz 0.10.0", "half", - "hashbrown 0.14.5", + "hashbrown 0.15.2", "num", ] [[package]] name = "arrow-buffer" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d0a2432f0cba5692bf4cb757469c66791394bac9ec7ce63c1afe74744c37b27" +checksum = "9e4a9b9b1d6d7117f6138e13bc4dd5daa7f94e671b70e8c9c4dc37b4f5ecfc16" dependencies = [ "bytes", "half", @@ -332,9 +332,9 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9abc10cd7995e83505cc290df9384d6e5412b207b79ce6bdff89a10505ed2cba" +checksum = "bc70e39916e60c5b7af7a8e2719e3ae589326039e1e863675a008bee5ffe90fd" dependencies = [ "arrow-array", "arrow-buffer", @@ -353,9 +353,9 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95cbcba196b862270bf2a5edb75927380a7f3a163622c61d40cbba416a6305f2" +checksum = "789b2af43c1049b03a8d088ff6b2257cdcea1756cd76b174b1f2600356771b97" dependencies = [ "arrow-array", "arrow-buffer", @@ -372,9 +372,9 @@ dependencies = [ [[package]] name = "arrow-data" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2742ac1f6650696ab08c88f6dd3f0eb68ce10f8c253958a18c943a68cd04aec5" +checksum = "e4e75edf21ffd53744a9b8e3ed11101f610e7ceb1a29860432824f1834a1f623" dependencies = [ "arrow-buffer", "arrow-schema", @@ -384,9 +384,9 @@ dependencies = [ [[package]] name = "arrow-flight" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3241ce691192d789b7b94f56a10e166ee608bdc3932c759eb0b85f09235352bb" +checksum = "4c09b331887a526f203f2123444792aee924632bd08b9940435070901075832e" dependencies = [ "arrow-array", "arrow-buffer", @@ -397,17 +397,17 @@ dependencies = [ "bytes", "futures", "paste", - "prost 0.12.6", - "prost-types 0.12.6", + "prost 0.13.3", + "prost-types 0.13.3", "tokio", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] name = "arrow-ipc" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a42ea853130f7e78b9b9d178cb4cd01dee0f78e64d96c2949dc0a915d6d9e19d" +checksum = "d186a909dece9160bf8312f5124d797884f608ef5435a36d9d608e0b2a9bcbf8" dependencies = [ "arrow-array", "arrow-buffer", @@ -421,9 +421,9 @@ dependencies = [ [[package]] name = "arrow-json" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eaafb5714d4e59feae964714d724f880511500e3569cc2a94d02456b403a2a49" +checksum = "b66ff2fedc1222942d0bd2fd391cb14a85baa3857be95c9373179bd616753b85" dependencies = [ "arrow-array", "arrow-buffer", @@ -441,9 +441,9 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3e6b61e3dc468f503181dccc2fc705bdcc5f2f146755fa5b56d0a6c5943f412" +checksum = "ece7b5bc1180e6d82d1a60e1688c199829e8842e38497563c3ab6ea813e527fd" dependencies = [ "arrow-array", "arrow-buffer", @@ -456,9 +456,9 @@ dependencies = [ [[package]] name = "arrow-row" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "848ee52bb92eb459b811fb471175ea3afcf620157674c8794f539838920f9228" +checksum = "745c114c8f0e8ce211c83389270de6fbe96a9088a7b32c2a041258a443fe83ff" dependencies = [ "ahash 0.8.11", "arrow-array", @@ -466,23 +466,22 @@ dependencies = [ "arrow-data", "arrow-schema", "half", - "hashbrown 0.14.5", ] [[package]] name = "arrow-schema" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "02d9483aaabe910c4781153ae1b6ae0393f72d9ef757d38d09d450070cf2e528" +checksum = "b95513080e728e4cec37f1ff5af4f12c9688d47795d17cda80b6ec2cf74d4678" dependencies = [ "serde", ] [[package]] name = "arrow-select" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "849524fa70e0e3c5ab58394c770cb8f514d0122d20de08475f7b472ed8075830" +checksum = "8e415279094ea70323c032c6e739c48ad8d80e78a09bef7117b8718ad5bf3722" dependencies = [ "ahash 0.8.11", "arrow-array", @@ -494,9 +493,9 @@ dependencies = [ [[package]] name = "arrow-string" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9373cb5a021aee58863498c37eb484998ef13377f69989c6c5ccfbd258236cdb" +checksum = "11d956cae7002eb8d83a27dbd34daaea1cf5b75852f0b84deb4d93a276e92bbf" dependencies = [ "arrow-array", "arrow-buffer", @@ -574,7 +573,7 @@ version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e614738943d3f68c628ae3dbce7c3daffb196665f82f8c8ea6b65de73c79429" dependencies = [ - "brotli 7.0.0", + "brotli", "bzip2", "flate2", "futures-core", @@ -767,16 +766,71 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b829e4e32b91e643de6eafe82b1d90675f5874230191a4ffbc1b336dec4d6bf" dependencies = [ "async-trait", - "axum-core", + "axum-core 0.3.4", "bitflags 1.3.2", "bytes", "futures-util", - "headers 0.3.9", "http 0.2.12", "http-body 0.4.6", "hyper 0.14.30", "itoa", - "matchit", + "matchit 0.7.3", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "rustversion", + "serde", + "sync_wrapper 0.1.2", + "tower 0.4.13", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum" +version = "0.7.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" +dependencies = [ + "async-trait", + "axum-core 0.4.5", + "bytes", + "futures-util", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "itoa", + "matchit 0.7.3", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "rustversion", + "serde", + "sync_wrapper 1.0.1", + "tower 0.5.2", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d6fd624c75e18b3b4c6b9caf42b1afe24437daaee904069137d8bab077be8b8" +dependencies = [ + "axum-core 0.5.0", + "bytes", + "form_urlencoded", + "futures-util", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "hyper 1.4.1", + "hyper-util", + "itoa", + "matchit 0.8.4", "memchr", "mime", "multer", @@ -787,11 +841,12 @@ dependencies = [ "serde_json", "serde_path_to_error", "serde_urlencoded", - "sync_wrapper 0.1.2", + "sync_wrapper 1.0.1", "tokio", - "tower", + "tower 0.5.2", "tower-layer", "tower-service", + "tracing", ] [[package]] @@ -812,12 +867,73 @@ dependencies = [ ] [[package]] -name = "axum-macros" -version = "0.3.8" +name = "axum-core" +version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdca6a10ecad987bda04e95606ef85a5417dcaac1a78455242d72e031e2b6b62" +checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199" +dependencies = [ + "async-trait", + "bytes", + "futures-util", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "mime", + "pin-project-lite", + "rustversion", + "sync_wrapper 1.0.1", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum-core" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df1362f362fd16024ae199c1970ce98f9661bf5ef94b9808fee734bc3698b733" +dependencies = [ + "bytes", + "futures-util", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "mime", + "pin-project-lite", + "rustversion", + "sync_wrapper 1.0.1", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "axum-extra" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fc6f625a1f7705c6cf62d0d070794e94668988b1c38111baeec177c715f7b" +dependencies = [ + "axum 0.8.1", + "axum-core 0.5.0", + "bytes", + "futures-util", + "headers", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "mime", + "pin-project-lite", + "serde", + "tower 0.5.2", + "tower-layer", + "tower-service", +] + +[[package]] +name = "axum-macros" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57d123550fa8d071b7255cb0cc04dc302baa6c8c4a79f55701552684d8399bce" dependencies = [ - "heck 0.4.1", "proc-macro2", "quote", "syn 2.0.90", @@ -1059,17 +1175,6 @@ dependencies = [ "syn_derive", ] -[[package]] -name = "brotli" -version = "3.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d640d25bc63c50fb1f0b545ffd80207d2e10a4c965530809b40ba3386825c391" -dependencies = [ - "alloc-no-stdlib", - "alloc-stdlib", - "brotli-decompressor 2.5.1", -] - [[package]] name = "brotli" version = "7.0.0" @@ -1078,17 +1183,7 @@ checksum = "cc97b8f16f944bba54f0433f07e30be199b6dc2bd25937444bbad560bcea29bd" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", - "brotli-decompressor 4.0.1", -] - -[[package]] -name = "brotli-decompressor" -version = "2.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e2e4afe60d7dd600fdd3de8d0f08c2b7ec039712e3b6137ff98b7004e82de4f" -dependencies = [ - "alloc-no-stdlib", - "alloc-stdlib", + "brotli-decompressor", ] [[package]] @@ -1275,7 +1370,7 @@ dependencies = [ "common-telemetry", "common-time", "common-version", - "dashmap", + "dashmap 5.5.3", "datafusion", "datatypes", "futures", @@ -1435,6 +1530,17 @@ dependencies = [ "phf", ] +[[package]] +name = "chrono-tz" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd6dd8046d00723a59a2f8c5f295c515b9bb9a331ee4f8f3d4dd49e428acd3b6" +dependencies = [ + "chrono", + "chrono-tz-build 0.4.0", + "phf", +] + [[package]] name = "chrono-tz-build" version = "0.2.1" @@ -1457,6 +1563,16 @@ dependencies = [ "phf_codegen", ] +[[package]] +name = "chrono-tz-build" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e94fea34d77a245229e7746bd2beb786cd2a896f306ff491fb8cecb3074b10a7" +dependencies = [ + "parse-zoneinfo", + "phf_codegen", +] + [[package]] name = "chunked_transfer" version = "1.5.0" @@ -1662,7 +1778,7 @@ dependencies = [ "moka", "parking_lot 0.12.3", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "query", "rand", "serde_json", @@ -1671,7 +1787,7 @@ dependencies = [ "substrait 0.37.3", "tokio", "tokio-stream", - "tonic 0.11.0", + "tonic 0.12.3", "tracing", ] @@ -1755,7 +1871,7 @@ dependencies = [ "nu-ansi-term", "plugins", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "query", "rand", "regex", @@ -1776,7 +1892,7 @@ dependencies = [ "tikv-jemallocator", "tokio", "toml 0.8.19", - "tonic 0.11.0", + "tonic 0.12.3", "tracing-appender", ] @@ -1915,10 +2031,10 @@ dependencies = [ name = "common-error" version = "0.12.0" dependencies = [ - "http 0.2.12", + "http 1.1.0", "snafu 0.8.5", "strum 0.25.0", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] @@ -2006,16 +2122,18 @@ dependencies = [ "common-telemetry", "common-time", "criterion 0.4.0", - "dashmap", + "dashmap 5.5.3", "datatypes", "flatbuffers", + "hyper 1.4.1", + "hyper-util", "lazy_static", - "prost 0.12.6", + "prost 0.13.3", "rand", "snafu 0.8.5", "tokio", - "tonic 0.11.0", - "tower", + "tonic 0.12.3", + "tower 0.5.2", ] [[package]] @@ -2031,7 +2149,7 @@ dependencies = [ "common-time", "datatypes", "paste", - "prost 0.12.6", + "prost 0.13.3", "snafu 0.8.5", "store-api", "table", @@ -2105,7 +2223,7 @@ dependencies = [ "lazy_static", "moka", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "rand", "regex", "rskafka", @@ -2119,7 +2237,7 @@ dependencies = [ "table", "tokio", "tokio-postgres", - "tonic 0.11.0", + "tonic 0.12.3", "typetag", "uuid", ] @@ -2144,7 +2262,7 @@ dependencies = [ "common-error", "common-macro", "pprof 0.14.0", - "prost 0.12.6", + "prost 0.13.3", "snafu 0.8.5", "tokio", ] @@ -2203,7 +2321,7 @@ dependencies = [ "futures-util", "serde", "snafu 0.8.5", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "sqlparser_derive 0.1.1", "statrs", "store-api", @@ -2798,6 +2916,20 @@ dependencies = [ "parking_lot_core 0.9.10", ] +[[package]] +name = "dashmap" +version = "6.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" +dependencies = [ + "cfg-if", + "crossbeam-utils", + "hashbrown 0.14.5", + "lock_api", + "once_cell", + "parking_lot_core 0.9.10", +] + [[package]] name = "data-encoding" version = "2.6.0" @@ -2806,10 +2938,9 @@ checksum = "e8566979429cf69b49a5c740c60791108e86440e8be149bbea4fe54d2c32d6e2" [[package]] name = "datafusion" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ - "ahash 0.8.11", "arrow", "arrow-array", "arrow-ipc", @@ -2819,34 +2950,34 @@ dependencies = [ "bytes", "bzip2", "chrono", - "dashmap", + "dashmap 6.1.0", + "datafusion-catalog", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate", - "datafusion-functions-array", + "datafusion-functions-nested", + "datafusion-functions-table", + "datafusion-functions-window", "datafusion-optimizer", "datafusion-physical-expr", "datafusion-physical-expr-common", + "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-sql", "flate2", "futures", "glob", - "half", - "hashbrown 0.14.5", - "indexmap 2.6.0", - "itertools 0.12.1", + "itertools 0.13.0", "log", - "num_cpus", "object_store", "parking_lot 0.12.3", "parquet", - "pin-project-lite", + "paste", "rand", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", + "sqlparser 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", "tempfile", "tokio", "tokio-util", @@ -2856,10 +2987,24 @@ dependencies = [ "zstd 0.13.2", ] +[[package]] +name = "datafusion-catalog" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "arrow-schema", + "async-trait", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-physical-plan", + "parking_lot 0.12.3", +] + [[package]] name = "datafusion-common" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "ahash 0.8.11", "arrow", @@ -2869,34 +3014,42 @@ dependencies = [ "chrono", "half", "hashbrown 0.14.5", - "instant", + "indexmap 2.6.0", "libc", - "num_cpus", "object_store", "parquet", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", + "paste", + "recursive", + "sqlparser 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", + "tokio", + "web-time 1.1.0", ] [[package]] name = "datafusion-common-runtime" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ + "log", "tokio", ] +[[package]] +name = "datafusion-doc" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" + [[package]] name = "datafusion-execution" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", "chrono", - "dashmap", + "dashmap 6.1.0", "datafusion-common", "datafusion-expr", "futures", - "hashbrown 0.14.5", "log", "object_store", "parking_lot 0.12.3", @@ -2907,38 +3060,59 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "ahash 0.8.11", "arrow", "arrow-array", + "arrow-buffer", "chrono", "datafusion-common", + "datafusion-doc", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", + "datafusion-functions-window-common", + "datafusion-physical-expr-common", + "indexmap 2.6.0", "paste", + "recursive", "serde_json", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", + "sqlparser 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", "strum 0.26.3", "strum_macros 0.26.4", ] [[package]] -name = "datafusion-functions" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +name = "datafusion-expr-common" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", + "datafusion-common", + "itertools 0.13.0", + "paste", +] + +[[package]] +name = "datafusion-functions" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "arrow", + "arrow-buffer", "base64 0.22.1", "blake2", "blake3", "chrono", "datafusion-common", + "datafusion-doc", "datafusion-execution", "datafusion-expr", - "datafusion-physical-expr", + "datafusion-macros", "hashbrown 0.14.5", "hex", - "itertools 0.12.1", + "itertools 0.13.0", "log", "md-5", "rand", @@ -2950,25 +3124,42 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "ahash 0.8.11", "arrow", "arrow-schema", "datafusion-common", + "datafusion-doc", "datafusion-execution", "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-macros", + "datafusion-physical-expr", "datafusion-physical-expr-common", + "half", "log", "paste", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] -name = "datafusion-functions-array" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +name = "datafusion-functions-aggregate-common" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "ahash 0.8.11", + "arrow", + "datafusion-common", + "datafusion-expr-common", + "datafusion-physical-expr-common", + "rand", +] + +[[package]] +name = "datafusion-functions-nested" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", "arrow-array", @@ -2979,15 +3170,76 @@ dependencies = [ "datafusion-execution", "datafusion-expr", "datafusion-functions", - "itertools 0.12.1", + "datafusion-functions-aggregate", + "datafusion-physical-expr-common", + "itertools 0.13.0", + "log", + "paste", + "rand", +] + +[[package]] +name = "datafusion-functions-table" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "ahash 0.8.11", + "arrow", + "arrow-schema", + "async-trait", + "datafusion-catalog", + "datafusion-common", + "datafusion-execution", + "datafusion-expr", + "datafusion-functions-aggregate-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", + "datafusion-physical-plan", + "half", + "indexmap 2.6.0", + "log", + "parking_lot 0.12.3", + "paste", +] + +[[package]] +name = "datafusion-functions-window" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "datafusion-common", + "datafusion-expr", + "datafusion-functions-window-common", + "datafusion-physical-expr", + "datafusion-physical-expr-common", "log", "paste", ] +[[package]] +name = "datafusion-functions-window-common" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "datafusion-common", + "datafusion-physical-expr-common", +] + +[[package]] +name = "datafusion-macros" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "datafusion-doc", + "proc-macro2", + "quote", + "syn 2.0.90", +] + [[package]] name = "datafusion-optimizer" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", "async-trait", @@ -2995,58 +3247,71 @@ dependencies = [ "datafusion-common", "datafusion-expr", "datafusion-physical-expr", - "hashbrown 0.14.5", "indexmap 2.6.0", - "itertools 0.12.1", + "itertools 0.13.0", "log", + "recursive", + "regex", "regex-syntax 0.8.5", ] [[package]] name = "datafusion-physical-expr" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "ahash 0.8.11", "arrow", "arrow-array", "arrow-buffer", - "arrow-ord", "arrow-schema", - "arrow-string", - "base64 0.22.1", - "chrono", "datafusion-common", - "datafusion-execution", "datafusion-expr", - "datafusion-functions-aggregate", + "datafusion-expr-common", + "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", "half", "hashbrown 0.14.5", - "hex", "indexmap 2.6.0", - "itertools 0.12.1", + "itertools 0.13.0", "log", "paste", "petgraph", - "regex", ] [[package]] name = "datafusion-physical-expr-common" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" +dependencies = [ + "ahash 0.8.11", + "arrow", + "datafusion-common", + "datafusion-expr-common", + "hashbrown 0.14.5", + "itertools 0.13.0", +] + +[[package]] +name = "datafusion-physical-optimizer" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", "datafusion-common", - "datafusion-expr", - "rand", + "datafusion-execution", + "datafusion-expr-common", + "datafusion-physical-expr", + "datafusion-physical-plan", + "itertools 0.13.0", + "log", + "recursive", ] [[package]] name = "datafusion-physical-plan" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "ahash 0.8.11", "arrow", @@ -3060,14 +3325,14 @@ dependencies = [ "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", - "datafusion-functions-aggregate", + "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "futures", "half", "hashbrown 0.14.5", "indexmap 2.6.0", - "itertools 0.12.1", + "itertools 0.13.0", "log", "once_cell", "parking_lot 0.12.3", @@ -3078,33 +3343,37 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ "arrow", "arrow-array", "arrow-schema", "datafusion-common", "datafusion-expr", + "indexmap 2.6.0", "log", + "recursive", "regex", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", - "strum 0.26.3", + "sqlparser 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] name = "datafusion-substrait" -version = "38.0.0" -source = "git+https://github.com/waynexia/arrow-datafusion.git?rev=7823ef2f63663907edab46af0d51359900f608d6#7823ef2f63663907edab46af0d51359900f608d6" +version = "43.0.0" +source = "git+https://github.com/apache/datafusion.git?rev=2464703c84c400a09cc59277018813f0e797bb4e#2464703c84c400a09cc59277018813f0e797bb4e" dependencies = [ + "arrow-buffer", "async-recursion", + "async-trait", "chrono", "datafusion", - "itertools 0.12.1", + "itertools 0.13.0", "object_store", "pbjson-types", - "prost 0.12.6", - "substrait 0.34.1", + "prost 0.13.3", + "substrait 0.49.5", + "url", ] [[package]] @@ -3135,7 +3404,7 @@ dependencies = [ "common-time", "common-version", "common-wal", - "dashmap", + "dashmap 5.5.3", "datafusion", "datafusion-common", "datafusion-expr", @@ -3151,7 +3420,7 @@ dependencies = [ "mito2", "object-store", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "query", "reqwest", "serde", @@ -3164,7 +3433,7 @@ dependencies = [ "table", "tokio", "toml 0.8.19", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] @@ -3192,7 +3461,7 @@ dependencies = [ "serde", "serde_json", "snafu 0.8.5", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "sqlparser_derive 0.1.1", ] @@ -3497,17 +3766,6 @@ dependencies = [ "winapi", ] -[[package]] -name = "displaydoc" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", -] - [[package]] name = "dlv-list" version = "0.3.0" @@ -3695,17 +3953,17 @@ dependencies = [ [[package]] name = "etcd-client" -version = "0.13.0" +version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b915bb9b1e143ab7062e0067ed663e3dfeffc69ce0ceb9e93b35fecfc158d28" +checksum = "39bde3ce50a626efeb1caa9ab1083972d178bebb55ca627639c8ded507dfcbde" dependencies = [ - "http 0.2.12", - "prost 0.12.6", + "http 1.1.0", + "prost 0.13.3", "tokio", "tokio-stream", - "tonic 0.11.0", - "tonic-build", - "tower", + "tonic 0.12.3", + "tonic-build 0.12.3", + "tower 0.4.13", "tower-service", ] @@ -3872,9 +4130,9 @@ checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" [[package]] name = "flatbuffers" -version = "23.5.26" +version = "24.3.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4dac53e22462d78c16d64a1cd22371b54cc3fe94aa15e7886a2fa6e5d1ab8640" +checksum = "8add37afff2d4ffa83bc748a70b4b1370984f6980768554182424ef71447c35f" dependencies = [ "bitflags 1.3.2", "rustc_version", @@ -3955,13 +4213,14 @@ dependencies = [ "datafusion-common", "datafusion-expr", "datafusion-physical-expr", + "datafusion-substrait", "datatypes", "enum-as-inner", "enum_dispatch", "futures", "get-size2", "greptime-proto", - "http 0.2.12", + "http 1.1.0", "hydroflow", "itertools 0.10.5", "lazy_static", @@ -3972,7 +4231,7 @@ dependencies = [ "partition", "pretty_assertions", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "query", "serde", "serde_json", @@ -3985,7 +4244,7 @@ dependencies = [ "substrait 0.12.0", "table", "tokio", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] @@ -4073,13 +4332,13 @@ dependencies = [ "log-query", "log-store", "meta-client", - "opentelemetry-proto 0.5.0", + "opentelemetry-proto 0.27.0", "operator", "partition", "pipeline", "prometheus", "promql-parser", - "prost 0.12.6", + "prost 0.13.3", "query", "raft-engine", "serde", @@ -4088,14 +4347,14 @@ dependencies = [ "session", "snafu 0.8.5", "sql", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "store-api", "strfmt", "table", "tokio", "toml 0.8.19", - "tonic 0.11.0", - "tower", + "tonic 0.12.3", + "tower 0.5.2", "uuid", ] @@ -4385,9 +4644,9 @@ dependencies = [ [[package]] name = "get-size2" -version = "0.1.2" +version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "159c430715e540d2198fa981d39cd45563ccc60900de187f5b152b33b1cb408e" +checksum = "3aa3d1f2527cf956b5637a531e21eb1ef9c825c70cd6f8765fd00b7457eef699" [[package]] name = "getopts" @@ -4451,15 +4710,15 @@ dependencies = [ [[package]] name = "greptime-proto" version = "0.1.0" -source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=6cee3db98a552f1dd848dec3eefcce8f26343748#6cee3db98a552f1dd848dec3eefcce8f26343748" +source = "git+https://github.com/GreptimeTeam/greptime-proto.git?rev=683e9d10ae7f3dfb8aaabd89082fc600c17e3795#683e9d10ae7f3dfb8aaabd89082fc600c17e3795" dependencies = [ - "prost 0.12.6", + "prost 0.13.3", "serde", "serde_json", "strum 0.25.0", "strum_macros 0.25.3", - "tonic 0.11.0", - "tonic-build", + "tonic 0.12.3", + "tonic-build 0.12.3", ] [[package]] @@ -4481,6 +4740,25 @@ dependencies = [ "tracing", ] +[[package]] +name = "h2" +version = "0.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccae279728d634d083c00f6099cb58f01cc99c145b84b8be2f6c74618d79922e" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http 1.1.0", + "indexmap 2.6.0", + "slab", + "tokio", + "tokio-util", + "tracing", +] + [[package]] name = "h3o" version = "0.6.4" @@ -4572,21 +4850,6 @@ dependencies = [ "num-traits", ] -[[package]] -name = "headers" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06683b93020a07e3dbcf5f8c0f6d40080d725bea7936fc01ad345c01b97dc270" -dependencies = [ - "base64 0.21.7", - "bytes", - "headers-core 0.2.0", - "http 0.2.12", - "httpdate", - "mime", - "sha1", -] - [[package]] name = "headers" version = "0.4.0" @@ -4595,22 +4858,13 @@ checksum = "322106e6bd0cba2d5ead589ddb8150a13d7c4217cf80d7c4f682ca994ccc6aa9" dependencies = [ "base64 0.21.7", "bytes", - "headers-core 0.3.0", + "headers-core", "http 1.1.0", "httpdate", "mime", "sha1", ] -[[package]] -name = "headers-core" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e7f66481bfee273957b1f20485a4ff3362987f85b2c236580d81b4eb7a326429" -dependencies = [ - "http 0.2.12", -] - [[package]] name = "headers-core" version = "0.3.0" @@ -4780,9 +5034,9 @@ dependencies = [ [[package]] name = "http-range-header" -version = "0.3.1" +version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "add0ab9360ddbd88cfeb3bd9574a1d85cfdfa14db10b3e21d3700dbc4328758f" +checksum = "9171a2ea8a68358193d15dd5d70c1c10a2afc3e7e4c5bc92bc9f025cebd7359c" [[package]] name = "httparse" @@ -4930,7 +5184,7 @@ dependencies = [ "futures-channel", "futures-core", "futures-util", - "h2", + "h2 0.3.26", "http 0.2.12", "http-body 0.4.6", "httparse", @@ -4953,9 +5207,11 @@ dependencies = [ "bytes", "futures-channel", "futures-util", + "h2 0.4.7", "http 1.1.0", "http-body 1.0.1", "httparse", + "httpdate", "itoa", "pin-project-lite", "smallvec", @@ -4971,7 +5227,7 @@ checksum = "5d06dbdfbacf34d996c6fb540a71a684a7aae9056c71951163af8a8a4c07b9a4" dependencies = [ "bytes", "futures-util", - "headers 0.4.0", + "headers", "http 1.1.0", "hyper 1.4.1", "hyper-rustls", @@ -5113,124 +5369,6 @@ dependencies = [ "cc", ] -[[package]] -name = "icu_collections" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db2fa452206ebee18c4b5c2274dbf1de17008e874b4dc4f0aea9d01ca79e4526" -dependencies = [ - "displaydoc", - "yoke", - "zerofrom", - "zerovec", -] - -[[package]] -name = "icu_locid" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13acbb8371917fc971be86fc8057c41a64b521c184808a698c02acc242dbf637" -dependencies = [ - "displaydoc", - "litemap", - "tinystr", - "writeable", - "zerovec", -] - -[[package]] -name = "icu_locid_transform" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "01d11ac35de8e40fdeda00d9e1e9d92525f3f9d887cdd7aa81d727596788b54e" -dependencies = [ - "displaydoc", - "icu_locid", - "icu_locid_transform_data", - "icu_provider", - "tinystr", - "zerovec", -] - -[[package]] -name = "icu_locid_transform_data" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fdc8ff3388f852bede6b579ad4e978ab004f139284d7b28715f773507b946f6e" - -[[package]] -name = "icu_normalizer" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19ce3e0da2ec68599d193c93d088142efd7f9c5d6fc9b803774855747dc6a84f" -dependencies = [ - "displaydoc", - "icu_collections", - "icu_normalizer_data", - "icu_properties", - "icu_provider", - "smallvec", - "utf16_iter", - "utf8_iter", - "write16", - "zerovec", -] - -[[package]] -name = "icu_normalizer_data" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8cafbf7aa791e9b22bec55a167906f9e1215fd475cd22adfcf660e03e989516" - -[[package]] -name = "icu_properties" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93d6020766cfc6302c15dbbc9c8778c37e62c14427cb7f6e601d849e092aeef5" -dependencies = [ - "displaydoc", - "icu_collections", - "icu_locid_transform", - "icu_properties_data", - "icu_provider", - "tinystr", - "zerovec", -] - -[[package]] -name = "icu_properties_data" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67a8effbc3dd3e4ba1afa8ad918d5684b8868b3b26500753effea8d2eed19569" - -[[package]] -name = "icu_provider" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ed421c8a8ef78d3e2dbc98a973be2f3770cb42b606e3ab18d6237c4dfde68d9" -dependencies = [ - "displaydoc", - "icu_locid", - "icu_provider_macros", - "stable_deref_trait", - "tinystr", - "writeable", - "yoke", - "zerofrom", - "zerovec", -] - -[[package]] -name = "icu_provider_macros" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ec89e9337638ecdc08744df490b221a7399bf8d164eb52a665454e60e075ad6" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", -] - [[package]] name = "ident_case" version = "1.0.1" @@ -5239,23 +5377,12 @@ checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" [[package]] name = "idna" -version = "1.0.3" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "686f825264d630750a544639377bae737628043f20d38bbc029e8f29ea968a7e" +checksum = "634d9b1461af396cad843f47fdba5597a4f9e6ddd4bfb6ff5d85028c25cb12f6" dependencies = [ - "idna_adapter", - "smallvec", - "utf8_iter", -] - -[[package]] -name = "idna_adapter" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "daca1df1c957320b2cf139ac61e7bd64fed304c5040df000a745aa1de3b4ef71" -dependencies = [ - "icu_normalizer", - "icu_properties", + "unicode-bidi", + "unicode-normalization", ] [[package]] @@ -5305,7 +5432,7 @@ dependencies = [ "jieba-rs", "mockall", "pin-project", - "prost 0.12.6", + "prost 0.13.3", "puffin", "rand", "regex", @@ -5766,8 +5893,8 @@ dependencies = [ "thiserror 1.0.64", "tokio", "tokio-util", - "tower", - "tower-http 0.5.2", + "tower 0.4.13", + "tower-http", "tracing", ] @@ -5879,9 +6006,9 @@ checksum = "0c2cdeb66e45e9f36bfad5bbdb4d2384e70936afbee843c6f6543f0c551ebb25" [[package]] name = "lexical-core" -version = "0.8.5" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2cde5de06e8d4c2faabc400238f9ae1c74d5412d03a7bd067645ccbc47070e46" +checksum = "0431c65b318a590c1de6b8fd6e72798c92291d27762d94c9e6c37ed7a73d8458" dependencies = [ "lexical-parse-float", "lexical-parse-integer", @@ -5892,9 +6019,9 @@ dependencies = [ [[package]] name = "lexical-parse-float" -version = "0.8.5" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "683b3a5ebd0130b8fb52ba0bdc718cc56815b6a097e28ae5a6997d0ad17dc05f" +checksum = "eb17a4bdb9b418051aa59d41d65b1c9be5affab314a872e5ad7f06231fb3b4e0" dependencies = [ "lexical-parse-integer", "lexical-util", @@ -5903,9 +6030,9 @@ dependencies = [ [[package]] name = "lexical-parse-integer" -version = "0.8.6" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d0994485ed0c312f6d965766754ea177d07f9c00c9b82a5ee62ed5b47945ee9" +checksum = "5df98f4a4ab53bf8b175b363a34c7af608fe31f93cc1fb1bf07130622ca4ef61" dependencies = [ "lexical-util", "static_assertions", @@ -5913,18 +6040,18 @@ dependencies = [ [[package]] name = "lexical-util" -version = "0.8.5" +version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5255b9ff16ff898710eb9eb63cb39248ea8a5bb036bea8085b1a767ff6c4e3fc" +checksum = "85314db53332e5c192b6bca611fb10c114a80d1b831ddac0af1e9be1b9232ca0" dependencies = [ "static_assertions", ] [[package]] name = "lexical-write-float" -version = "0.8.5" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accabaa1c4581f05a3923d1b4cfd124c329352288b7b9da09e766b0668116862" +checksum = "6e7c3ad4e37db81c1cbe7cf34610340adc09c322871972f74877a712abc6c809" dependencies = [ "lexical-util", "lexical-write-integer", @@ -5933,9 +6060,9 @@ dependencies = [ [[package]] name = "lexical-write-integer" -version = "0.8.5" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1b6f3d1f4422866b68192d62f77bc5c700bee84f3069f2469d7bc8c77852446" +checksum = "eb89e9f6958b83258afa3deed90b5de9ef68eef090ad5086c791cd2345610162" dependencies = [ "lexical-util", "static_assertions", @@ -5943,9 +6070,9 @@ dependencies = [ [[package]] name = "libc" -version = "0.2.159" +version = "0.2.169" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "561d97a539a36e26a9a5fad1ea11a3039a67714694aaa379433e580854bc3dc5" +checksum = "b5aba8db14291edd000dfcc4d620c7ebfb122c613afb886ca8803fa4e128a20a" [[package]] name = "libfuzzer-sys" @@ -6041,12 +6168,6 @@ version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78b3ae25bc7c8c38cec158d1f2757ee79e9b3740fbc7ccf0e59e4b08d793fa89" -[[package]] -name = "litemap" -version = "0.7.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ee93343901ab17bd981295f2cf0026d4ad018c7c31ba84549a4ddbb47a45104" - [[package]] name = "local-ip-address" version = "0.6.3" @@ -6130,11 +6251,10 @@ dependencies = [ [[package]] name = "loki-api" version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "674883a98273598ac3aad4301724c56734bea90574c5033af067e8f9fb5eb399" +source = "git+https://github.com/shuiyisong/tracing-loki?branch=chore%2Fprost_version#ad135292a77b6ce638d126571804c19fdaa36215" dependencies = [ - "prost 0.12.6", - "prost-types 0.12.6", + "prost 0.13.3", + "prost-types 0.13.3", ] [[package]] @@ -6279,6 +6399,12 @@ version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94" +[[package]] +name = "matchit" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" + [[package]] name = "matrixmultiply" version = "0.3.9" @@ -6380,8 +6506,8 @@ dependencies = [ "snafu 0.8.5", "tokio", "tokio-stream", - "tonic 0.11.0", - "tower", + "tonic 0.12.3", + "tower 0.5.2", "tracing", "tracing-subscriber", ] @@ -6392,6 +6518,7 @@ version = "0.12.0" dependencies = [ "api", "async-trait", + "bytes", "chrono", "clap 4.5.19", "client", @@ -6411,24 +6538,25 @@ dependencies = [ "common-time", "common-version", "common-wal", - "dashmap", + "dashmap 5.5.3", "datatypes", "deadpool", "deadpool-postgres", "derive_builder 0.12.0", "etcd-client", "futures", - "h2", - "http-body 0.4.6", + "h2 0.3.26", + "http-body-util", "humantime", "humantime-serde", + "hyper-util", "itertools 0.10.5", "lazy_static", "local-ip-address", "once_cell", "parking_lot 0.12.3", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "rand", "regex", "serde", @@ -6442,8 +6570,8 @@ dependencies = [ "tokio-postgres", "tokio-stream", "toml 0.8.19", - "tonic 0.11.0", - "tower", + "tonic 0.12.3", + "tower 0.5.2", "tracing", "tracing-subscriber", "typetag", @@ -6612,7 +6740,7 @@ dependencies = [ "paste", "pin-project", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "puffin", "rand", "regex", @@ -6689,16 +6817,15 @@ dependencies = [ [[package]] name = "multer" -version = "2.1.0" +version = "3.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "01acbdc23469fd8fe07ab135923371d5f5a422fbf9c522158677c8eb15bc51c2" +checksum = "83e87776546dc87511aa5ee218730c92b666d7264ab6ed41f9d215af9cd5224b" dependencies = [ "bytes", "encoding_rs", "futures-util", - "http 0.2.12", + "http 1.1.0", "httparse", - "log", "memchr", "mime", "spin", @@ -6976,9 +7103,9 @@ checksum = "27b02d87554356db9e9a873add8782d4ea6e3e58ea071a9adb9a2e8ddb884a8b" [[package]] name = "neli" -version = "0.6.4" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1100229e06604150b3becd61a4965d5c70f3be1759544ea7274166f4be41ef43" +checksum = "93062a0dce6da2517ea35f301dfc88184ce18d3601ec786a727a87bf535deca9" dependencies = [ "byteorder", "libc", @@ -6988,9 +7115,9 @@ dependencies = [ [[package]] name = "neli-proc-macros" -version = "0.1.3" +version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c168194d373b1e134786274020dae7fc5513d565ea2ebb9bc9ff17ffb69106d4" +checksum = "0c8034b7fbb6f9455b2a96c19e6edf8dc9fc34c70449938d8ee3b4df363f61fe" dependencies = [ "either", "proc-macro2", @@ -7285,19 +7412,19 @@ dependencies = [ [[package]] name = "object_store" -version = "0.9.1" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8718f8b65fdf67a45108d1548347d4af7d71fb81ce727bbf9e3b2535e079db3" +checksum = "6eb4c22c6154a1e759d7099f9ffad7cc5ef8245f9efbab4a41b92623079c82f3" dependencies = [ "async-trait", "bytes", "chrono", "futures", "humantime", - "itertools 0.12.1", + "itertools 0.13.0", "parking_lot 0.12.3", "percent-encoding", - "snafu 0.7.5", + "snafu 0.8.5", "tokio", "tracing", "url", @@ -7408,17 +7535,16 @@ dependencies = [ [[package]] name = "opentelemetry" -version = "0.22.0" +version = "0.27.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "900d57987be3f2aeb70d385fff9b27fb74c5723cc9a52d904d4f9c807a0667bf" +checksum = "ab70038c28ed37b97d8ed414b6429d343a8bbf44c9f79ec854f3a643029ba6d7" dependencies = [ "futures-core", "futures-sink", "js-sys", - "once_cell", "pin-project-lite", "thiserror 1.0.64", - "urlencoding", + "tracing", ] [[package]] @@ -7454,16 +7580,16 @@ dependencies = [ [[package]] name = "opentelemetry-proto" -version = "0.5.0" +version = "0.27.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a8fddc9b68f5b80dae9d6f510b88e02396f006ad48cac349411fbecc80caae4" +checksum = "a6e05acbfada5ec79023c85368af14abd0b307c015e9064d249b2a950ef459a6" dependencies = [ "hex", - "opentelemetry 0.22.0", - "opentelemetry_sdk 0.22.1", - "prost 0.12.6", + "opentelemetry 0.27.1", + "opentelemetry_sdk 0.27.1", + "prost 0.13.3", "serde", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] @@ -7499,19 +7625,16 @@ dependencies = [ [[package]] name = "opentelemetry_sdk" -version = "0.22.1" +version = "0.27.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e90c7113be649e31e9a0f8b5ee24ed7a16923b322c3c5ab6367469c049d6b7e" +checksum = "231e9d6ceef9b0b2546ddf52335785ce41252bc7474ee8ba05bfad277be13ab8" dependencies = [ "async-trait", - "crossbeam-channel", "futures-channel", "futures-executor", "futures-util", "glob", - "once_cell", - "opentelemetry 0.22.0", - "ordered-float 4.3.0", + "opentelemetry 0.27.1", "percent-encoding", "rand", "serde_json", @@ -7566,25 +7689,27 @@ dependencies = [ "session", "snafu 0.8.5", "sql", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "store-api", "substrait 0.12.0", "table", "tokio", "tokio-util", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] name = "orc-rust" -version = "0.3.0" -source = "git+https://github.com/datafusion-contrib/datafusion-orc.git?rev=502217315726314c4008808fe169764529640599#502217315726314c4008808fe169764529640599" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b307d095f0481b043b27d94eeb95cd67a8b19c6a34685ad0ccd884740a9306e" dependencies = [ "arrow", "async-trait", + "bytemuck", "bytes", "chrono", - "chrono-tz 0.8.6", + "chrono-tz 0.10.0", "fallible-streaming-iterator", "flate2", "futures", @@ -7592,11 +7717,11 @@ dependencies = [ "lz4_flex", "lzokay-native", "num", - "prost 0.11.9", - "snafu 0.7.5", + "prost 0.13.3", + "snafu 0.8.5", "snap", "tokio", - "zstd 0.12.4", + "zstd 0.13.2", ] [[package]] @@ -7758,9 +7883,9 @@ dependencies = [ [[package]] name = "parquet" -version = "51.0.0" +version = "53.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "096795d4f47f65fd3ee1ec5a98b77ab26d602f2cc785b0e4be5443add17ecc32" +checksum = "2b449890367085eb65d7d3321540abc3d7babbd179ce31df0016e90719114191" dependencies = [ "ahash 0.8.11", "arrow-array", @@ -7771,13 +7896,13 @@ dependencies = [ "arrow-schema", "arrow-select", "base64 0.22.1", - "brotli 3.5.0", + "brotli", "bytes", "chrono", "flate2", "futures", "half", - "hashbrown 0.14.5", + "hashbrown 0.15.2", "lz4_flex", "num", "num-bigint", @@ -7789,6 +7914,7 @@ dependencies = [ "tokio", "twox-hash", "zstd 0.13.2", + "zstd-sys", ] [[package]] @@ -7818,7 +7944,7 @@ dependencies = [ "serde_json", "snafu 0.8.5", "sql", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "store-api", "table", ] @@ -7843,9 +7969,9 @@ checksum = "8835116a5c179084a830efb3adc117ab007512b535bc1a21c991d3b32a6b44dd" [[package]] name = "pbjson" -version = "0.6.0" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1030c719b0ec2a2d25a5df729d6cff1acf3cc230bf766f4f97833591f7577b90" +checksum = "c7e6349fa080353f4a597daffd05cb81572a9c031a6d4fff7e504947496fcc68" dependencies = [ "base64 0.21.7", "serde", @@ -7853,28 +7979,28 @@ dependencies = [ [[package]] name = "pbjson-build" -version = "0.6.2" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2580e33f2292d34be285c5bc3dba5259542b083cfad6037b6d70345f24dcb735" +checksum = "6eea3058763d6e656105d1403cb04e0a41b7bbac6362d413e7c33be0c32279c9" dependencies = [ - "heck 0.4.1", - "itertools 0.11.0", - "prost 0.12.6", - "prost-types 0.12.6", + "heck 0.5.0", + "itertools 0.13.0", + "prost 0.13.3", + "prost-types 0.13.3", ] [[package]] name = "pbjson-types" -version = "0.6.0" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18f596653ba4ac51bdecbb4ef6773bc7f56042dc13927910de1684ad3d32aa12" +checksum = "e54e5e7bfb1652f95bc361d76f3c780d8e526b134b85417e774166ee941f0887" dependencies = [ "bytes", "chrono", "pbjson", "pbjson-build", - "prost 0.12.6", - "prost-build 0.12.6", + "prost 0.13.3", + "prost-build 0.13.3", "serde", ] @@ -7992,7 +8118,7 @@ dependencies = [ "rand", "ring", "rust_decimal", - "thiserror 2.0.6", + "thiserror 2.0.11", "tokio", "tokio-rustls 0.26.0", "tokio-util", @@ -8092,7 +8218,7 @@ dependencies = [ "criterion 0.4.0", "crossbeam-utils", "csv", - "dashmap", + "dashmap 5.5.3", "datafusion", "datafusion-common", "datafusion-expr", @@ -8378,9 +8504,9 @@ dependencies = [ [[package]] name = "prettyplease" -version = "0.2.22" +version = "0.2.25" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "479cf940fbbb3426c32c5d5176f62ad57549a0bb84773423ba8be9d089f5faba" +checksum = "64d1ec885c64d0457d564db4ec299b2dae3f9c02808b8ad9c3a089c591b18033" dependencies = [ "proc-macro2", "syn 2.0.90", @@ -8508,7 +8634,7 @@ dependencies = [ "greptime-proto", "lazy_static", "prometheus", - "prost 0.12.6", + "prost 0.13.3", "snafu 0.8.5", "tokio", ] @@ -8593,9 +8719,11 @@ dependencies = [ "multimap", "once_cell", "petgraph", + "prettyplease", "prost 0.13.3", "prost-types 0.13.3", "regex", + "syn 2.0.90", "tempfile", ] @@ -8705,6 +8833,15 @@ dependencies = [ "autotools", ] +[[package]] +name = "psm" +version = "0.1.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "200b9ff220857e53e184257720a14553b2f4aa02577d2ed9842d45d4b9654810" +dependencies = [ + "cc", +] + [[package]] name = "ptr_meta" version = "0.1.4" @@ -8837,7 +8974,7 @@ dependencies = [ "prometheus", "promql", "promql-parser", - "prost 0.12.6", + "prost 0.13.3", "rand", "regex", "serde", @@ -8845,7 +8982,7 @@ dependencies = [ "session", "snafu 0.8.5", "sql", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "statrs", "store-api", "substrait 0.12.0", @@ -9090,6 +9227,26 @@ dependencies = [ "crossbeam-utils", ] +[[package]] +name = "recursive" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0786a43debb760f491b1bc0269fe5e84155353c67482b9e60d0cfb596054b43e" +dependencies = [ + "recursive-proc-macro-impl", + "stacker", +] + +[[package]] +name = "recursive-proc-macro-impl" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" +dependencies = [ + "quote", + "syn 2.0.90", +] + [[package]] name = "redox_syscall" version = "0.2.16" @@ -9199,6 +9356,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "regress" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1541daf4e4ed43a0922b7969bdc2170178bcacc5dabf7e39bc508a9fa3953a7a" +dependencies = [ + "hashbrown 0.14.5", + "memchr", +] + [[package]] name = "relative-path" version = "1.9.3" @@ -9247,9 +9414,9 @@ dependencies = [ [[package]] name = "reqwest" -version = "0.12.9" +version = "0.12.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a77c62af46e79de0a562e1a9849205ffcb7fc1238876e9bd743357570e04046f" +checksum = "f713147fbe92361e52392c73b8c9e48c04c6625bce969ef54dc901e58e042a7b" dependencies = [ "base64 0.22.1", "bytes", @@ -9672,20 +9839,6 @@ dependencies = [ "sct", ] -[[package]] -name = "rustls" -version = "0.22.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf4ef73721ac7bcd79b2b315da7779d8fc09718c6b3d2d1b2d94850eb8c18432" -dependencies = [ - "log", - "ring", - "rustls-pki-types", - "rustls-webpki 0.102.8", - "subtle", - "zeroize", -] - [[package]] name = "rustls" version = "0.23.20" @@ -10028,9 +10181,9 @@ checksum = "a3f0bf26fd526d2a95683cd0f87bf103b8539e2ca1ef48ce002d67aad59aa0b4" [[package]] name = "serde" -version = "1.0.210" +version = "1.0.215" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8e3592472072e6e22e0a54d5904d9febf8508f65fb8552499a1abc7d1078c3a" +checksum = "6513c1ad0b11a9376da888e3e0baa0077f1aed55c17f50e7b2397136129fb88f" dependencies = [ "serde_derive", ] @@ -10047,9 +10200,9 @@ dependencies = [ [[package]] name = "serde_derive" -version = "1.0.210" +version = "1.0.215" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "243902eda00fad750862fc144cea25caca5e20d615af0a81bee94ca738f1df1f" +checksum = "ad1e866f866923f252f05c889987993144fb74e722403468a4ebd70c3cd756c0" dependencies = [ "proc-macro2", "quote", @@ -10069,9 +10222,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.128" +version = "1.0.133" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ff5456707a1de34e7e37f2a6fd3d3f808c318259cbd01ab6377795054b483d8" +checksum = "c7fceb2473b9166b2294ef05efcb65a3db80803f0b03ef86a5fc88a2b85ee377" dependencies = [ "indexmap 2.6.0", "itoa", @@ -10189,7 +10342,8 @@ dependencies = [ "arrow-schema", "async-trait", "auth", - "axum", + "axum 0.8.1", + "axum-extra", "axum-macros", "base64 0.21.7", "bytes", @@ -10214,7 +10368,7 @@ dependencies = [ "common-time", "common-version", "criterion 0.5.1", - "dashmap", + "dashmap 5.5.3", "datafusion", "datafusion-common", "datafusion-expr", @@ -10223,13 +10377,13 @@ dependencies = [ "futures", "futures-util", "hashbrown 0.15.2", - "headers 0.3.9", + "headers", "hostname", - "http 0.2.12", - "http-body 0.4.6", + "http 1.1.0", + "http-body 1.0.1", "humantime", "humantime-serde", - "hyper 0.14.30", + "hyper 1.4.1", "influxdb_line_protocol", "itertools 0.10.5", "json5", @@ -10245,7 +10399,7 @@ dependencies = [ "once_cell", "openmetrics-parser", "opensrv-mysql", - "opentelemetry-proto 0.5.0", + "opentelemetry-proto 0.27.0", "parking_lot 0.12.3", "permutation", "pgwire", @@ -10255,7 +10409,7 @@ dependencies = [ "pprof 0.13.0", "prometheus", "promql-parser", - "prost 0.12.6", + "prost 0.13.3", "query", "quoted-string", "rand", @@ -10282,10 +10436,10 @@ dependencies = [ "tokio-rustls 0.26.0", "tokio-stream", "tokio-util", - "tonic 0.11.0", + "tonic 0.12.3", "tonic-reflection", - "tower", - "tower-http 0.4.4", + "tower 0.5.2", + "tower-http", "urlencoding", "uuid", "zstd 0.13.2", @@ -10630,7 +10784,7 @@ dependencies = [ "serde", "serde_json", "snafu 0.8.5", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "sqlparser_derive 0.1.1", "store-api", "table", @@ -10685,9 +10839,9 @@ dependencies = [ [[package]] name = "sqlparser" -version = "0.45.0" +version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f7bbffee862a796d67959a89859d6b1046bb5016d63e23835ad0da182777bbe0" +checksum = "9a875d8cd437cc8a97e9aeaeea352ec9a19aea99c23e9effb17757291de80b08" dependencies = [ "log", "sqlparser_derive 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", @@ -10695,15 +10849,15 @@ dependencies = [ [[package]] name = "sqlparser" -version = "0.45.0" -source = "git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7#54a267ac89c09b11c0c88934690530807185d3e7" +version = "0.52.0" +source = "git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170#71dd86058d2af97b9925093d40c4e03360403170" dependencies = [ "lazy_static", "log", "regex", "serde", - "sqlparser 0.45.0 (registry+https://github.com/rust-lang/crates.io-index)", - "sqlparser_derive 0.2.2 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", + "sqlparser_derive 0.2.2 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", ] [[package]] @@ -10731,7 +10885,7 @@ dependencies = [ [[package]] name = "sqlparser_derive" version = "0.2.2" -source = "git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7#54a267ac89c09b11c0c88934690530807185d3e7" +source = "git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170#71dd86058d2af97b9925093d40c4e03360403170" dependencies = [ "proc-macro2", "quote", @@ -10780,7 +10934,7 @@ dependencies = [ "serde_json", "sha2", "smallvec", - "thiserror 2.0.6", + "thiserror 2.0.11", "tokio", "tokio-stream", "tracing", @@ -10865,7 +11019,7 @@ dependencies = [ "smallvec", "sqlx-core", "stringprep", - "thiserror 2.0.6", + "thiserror 2.0.11", "tracing", "whoami", ] @@ -10903,7 +11057,7 @@ dependencies = [ "smallvec", "sqlx-core", "stringprep", - "thiserror 2.0.6", + "thiserror 2.0.11", "tracing", "whoami", ] @@ -10938,6 +11092,19 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3" +[[package]] +name = "stacker" +version = "0.1.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799c883d55abdb5e98af1a7b3f23b9b6de8ecada0ecac058672d7635eb48ca7b" +dependencies = [ + "cc", + "cfg-if", + "libc", + "psm", + "windows-sys 0.59.0", +] + [[package]] name = "static_assertions" version = "1.1.0" @@ -10978,7 +11145,7 @@ dependencies = [ "derive_builder 0.12.0", "futures", "humantime", - "prost 0.12.6", + "prost 0.13.3", "serde", "serde_json", "snafu 0.8.5", @@ -11102,36 +11269,12 @@ dependencies = [ "datafusion-substrait", "datatypes", "promql", - "prost 0.12.6", + "prost 0.13.3", "snafu 0.8.5", "substrait 0.37.3", "tokio", ] -[[package]] -name = "substrait" -version = "0.34.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04c77dec9b6c4e48ac828937bbe7cf473b0933168c5d76d51a5816ace7046be9" -dependencies = [ - "heck 0.5.0", - "pbjson", - "pbjson-build", - "pbjson-types", - "prettyplease", - "prost 0.12.6", - "prost-build 0.12.6", - "prost-types 0.12.6", - "schemars", - "semver", - "serde", - "serde_json", - "serde_yaml", - "syn 2.0.90", - "typify", - "walkdir", -] - [[package]] name = "substrait" version = "0.37.3" @@ -11149,7 +11292,32 @@ dependencies = [ "serde_json", "serde_yaml", "syn 2.0.90", - "typify", + "typify 0.1.0", + "walkdir", +] + +[[package]] +name = "substrait" +version = "0.49.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c271a596176d3b82bfc5b4107fe9fbd30e6a9a99c0dca146777f05d8f0e08e4" +dependencies = [ + "heck 0.5.0", + "pbjson", + "pbjson-build", + "pbjson-types", + "prettyplease", + "prost 0.13.3", + "prost-build 0.13.3", + "prost-types 0.13.3", + "regress 0.10.1", + "schemars", + "semver", + "serde", + "serde_json", + "serde_yaml", + "syn 2.0.90", + "typify 0.2.0", "walkdir", ] @@ -11241,17 +11409,6 @@ dependencies = [ "futures-core", ] -[[package]] -name = "synstructure" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8af7666ab7b6390ab78131fb5b0fce11d6b7a6951602017c35fa82800708971" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", -] - [[package]] name = "sysinfo" version = "0.30.13" @@ -11554,7 +11711,7 @@ dependencies = [ "serde_yaml", "snafu 0.8.5", "sql", - "sqlparser 0.45.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=54a267ac89c09b11c0c88934690530807185d3e7)", + "sqlparser 0.52.0 (git+https://github.com/GreptimeTeam/sqlparser-rs.git?rev=71dd86058d2af97b9925093d40c4e03360403170)", "sqlx", "store-api", "strum 0.25.0", @@ -11571,7 +11728,7 @@ dependencies = [ "async-stream", "async-trait", "auth", - "axum", + "axum 0.8.1", "cache", "catalog", "chrono", @@ -11603,6 +11760,7 @@ dependencies = [ "futures", "futures-util", "hex", + "hyper-util", "itertools 0.10.5", "loki-api", "meta-client", @@ -11610,11 +11768,11 @@ dependencies = [ "moka", "mysql_async", "object-store", - "opentelemetry-proto 0.5.0", + "opentelemetry-proto 0.27.0", "operator", "partition", "paste", - "prost 0.12.6", + "prost 0.13.3", "query", "rand", "rstest", @@ -11634,8 +11792,8 @@ dependencies = [ "tokio", "tokio-postgres", "tokio-stream", - "tonic 0.11.0", - "tower", + "tonic 0.12.3", + "tower 0.5.2", "url", "uuid", "zstd 0.13.2", @@ -11667,11 +11825,11 @@ dependencies = [ [[package]] name = "thiserror" -version = "2.0.6" +version = "2.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fec2a1820ebd077e2b90c4df007bebf344cd394098a13c563957d0afc83ea47" +checksum = "d452f284b73e6d76dd36758a0c8684b1d5be31f92b89d07fd5822175732206fc" dependencies = [ - "thiserror-impl 2.0.6", + "thiserror-impl 2.0.11", ] [[package]] @@ -11687,9 +11845,9 @@ dependencies = [ [[package]] name = "thiserror-impl" -version = "2.0.6" +version = "2.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d65750cab40f4ff1929fb1ba509e9914eb756131cef4210da8d5d700d26f6312" +checksum = "26afc1baea8a989337eeb52b6e72a039780ce45c3edfcc9c5b9d112feeb173c2" dependencies = [ "proc-macro2", "quote", @@ -11812,16 +11970,6 @@ dependencies = [ "log", ] -[[package]] -name = "tinystr" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9117f5d4db391c1cf6927e7bea3db74b9a1c1add8f7eda9ffd5364f40f57b82f" -dependencies = [ - "displaydoc", - "zerovec", -] - [[package]] name = "tinytemplate" version = "1.2.1" @@ -11849,9 +11997,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.42.0" +version = "1.40.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5cec9b21b0450273377fc97bd4c33a8acffc8c996c987a7c5b319a0083707551" +checksum = "e2b070231665d27ad9ec9b8df639893f46727666c6767db40317fbe920a5d998" dependencies = [ "backtrace", "bytes", @@ -11962,17 +12110,6 @@ dependencies = [ "tokio", ] -[[package]] -name = "tokio-rustls" -version = "0.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "775e0c0f0adb3a2f22a00c4745d728b479985fc15ee7ca6a2608388c5569860f" -dependencies = [ - "rustls 0.22.4", - "rustls-pki-types", - "tokio", -] - [[package]] name = "tokio-rustls" version = "0.26.0" @@ -12084,12 +12221,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3082666a3a6433f7f511c7192923fa1fe07c69332d3c6a2e6bb040b569199d5a" dependencies = [ "async-trait", - "axum", + "axum 0.6.20", "base64 0.21.7", "bytes", "futures-core", "futures-util", - "h2", + "h2 0.3.26", "http 0.2.12", "http-body 0.4.6", "hyper 0.14.30", @@ -12099,7 +12236,7 @@ dependencies = [ "prost 0.11.9", "tokio", "tokio-stream", - "tower", + "tower 0.4.13", "tower-layer", "tower-service", "tracing", @@ -12107,34 +12244,36 @@ dependencies = [ [[package]] name = "tonic" -version = "0.11.0" +version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76c4eb7a4e9ef9d4763600161f12f5070b92a578e1b634db88a6887844c91a13" +checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52" dependencies = [ "async-stream", "async-trait", - "axum", - "base64 0.21.7", + "axum 0.7.9", + "base64 0.22.1", "bytes", "flate2", - "h2", - "http 0.2.12", - "http-body 0.4.6", - "hyper 0.14.30", - "hyper-timeout 0.4.1", + "h2 0.4.7", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", + "hyper 1.4.1", + "hyper-timeout 0.5.1", + "hyper-util", "percent-encoding", "pin-project", - "prost 0.12.6", + "prost 0.13.3", "rustls-pemfile 2.2.0", - "rustls-pki-types", + "socket2", "tokio", - "tokio-rustls 0.25.0", + "tokio-rustls 0.26.0", "tokio-stream", - "tower", + "tower 0.4.13", "tower-layer", "tower-service", "tracing", - "zstd 0.12.4", + "zstd 0.13.2", ] [[package]] @@ -12151,16 +12290,30 @@ dependencies = [ ] [[package]] -name = "tonic-reflection" -version = "0.11.0" +name = "tonic-build" +version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "548c227bd5c0fae5925812c4ec6c66ffcfced23ea370cb823f4d18f0fc1cb6a7" +checksum = "9557ce109ea773b399c9b9e5dca39294110b74f1f342cb347a80d1fce8c26a11" dependencies = [ - "prost 0.12.6", - "prost-types 0.12.6", + "prettyplease", + "proc-macro2", + "prost-build 0.13.3", + "prost-types 0.13.3", + "quote", + "syn 2.0.90", +] + +[[package]] +name = "tonic-reflection" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "878d81f52e7fcfd80026b7fdb6a9b578b3c3653ba987f87f0dce4b64043cba27" +dependencies = [ + "prost 0.13.3", + "prost-types 0.13.3", "tokio", "tokio-stream", - "tonic 0.11.0", + "tonic 0.12.3", ] [[package]] @@ -12171,7 +12324,6 @@ checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c" dependencies = [ "futures-core", "futures-util", - "hdrhistogram", "indexmap 1.9.3", "pin-project", "pin-project-lite", @@ -12185,10 +12337,30 @@ dependencies = [ ] [[package]] -name = "tower-http" -version = "0.4.4" +name = "tower" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61c5bb1d698276a2443e5ecfabc1008bf15a36c12e6a7176e7bf089ea9131140" +checksum = "d039ad9159c98b70ecfd540b2573b97f7f52c3e8d9f8ad57a24b916a536975f9" +dependencies = [ + "futures-core", + "futures-util", + "hdrhistogram", + "indexmap 2.6.0", + "pin-project-lite", + "slab", + "sync_wrapper 1.0.1", + "tokio", + "tokio-util", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tower-http" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5" dependencies = [ "async-compression 0.4.13", "base64 0.21.7", @@ -12196,8 +12368,9 @@ dependencies = [ "bytes", "futures-core", "futures-util", - "http 0.2.12", - "http-body 0.4.6", + "http 1.1.0", + "http-body 1.0.1", + "http-body-util", "http-range-header", "httpdate", "iri-string", @@ -12207,32 +12380,13 @@ dependencies = [ "pin-project-lite", "tokio", "tokio-util", - "tower", + "tower 0.4.13", "tower-layer", "tower-service", "tracing", "uuid", ] -[[package]] -name = "tower-http" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5" -dependencies = [ - "base64 0.21.7", - "bitflags 2.6.0", - "bytes", - "http 1.1.0", - "http-body 1.0.1", - "http-body-util", - "mime", - "pin-project-lite", - "tower-layer", - "tower-service", - "tracing", -] - [[package]] name = "tower-layer" version = "0.3.3" @@ -12327,7 +12481,7 @@ dependencies = [ "tracing-core", "tracing-log 0.2.0", "tracing-subscriber", - "web-time", + "web-time 0.2.4", ] [[package]] @@ -12543,8 +12697,18 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adb6beec125971dda80a086f90b4a70f60f222990ce4d63ad0fc140492f53444" dependencies = [ - "typify-impl", - "typify-macro", + "typify-impl 0.1.0", + "typify-macro 0.1.0", +] + +[[package]] +name = "typify" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4c644dda9862f0fef3a570d8ddb3c2cfb1d5ac824a1f2ddfa7bc8f071a5ad8a" +dependencies = [ + "typify-impl 0.2.0", + "typify-macro 0.2.0", ] [[package]] @@ -12557,7 +12721,27 @@ dependencies = [ "log", "proc-macro2", "quote", - "regress", + "regress 0.9.1", + "schemars", + "semver", + "serde", + "serde_json", + "syn 2.0.90", + "thiserror 1.0.64", + "unicode-ident", +] + +[[package]] +name = "typify-impl" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d59ab345b6c0d8ae9500b9ff334a4c7c0d316c1c628dc55726b95887eb8dbd11" +dependencies = [ + "heck 0.5.0", + "log", + "proc-macro2", + "quote", + "regress 0.10.1", "schemars", "semver", "serde", @@ -12581,7 +12765,24 @@ dependencies = [ "serde_json", "serde_tokenstream", "syn 2.0.90", - "typify-impl", + "typify-impl 0.1.0", +] + +[[package]] +name = "typify-macro" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "785e2cdcef0df8160fdd762ed548a637aaec1e83704fdbc14da0df66013ee8d0" +dependencies = [ + "proc-macro2", + "quote", + "schemars", + "semver", + "serde", + "serde_json", + "serde_tokenstream", + "syn 2.0.90", + "typify-impl 0.2.0", ] [[package]] @@ -12687,9 +12888,9 @@ checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" [[package]] name = "url" -version = "2.5.4" +version = "2.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32f8b686cadd1473f4bd0117a5d28d36b1ade384ea9b5069a1c40aefed7fda60" +checksum = "22784dbdf76fdde8af1aeda5622b546b422b6fc585325248a2bf9f5e41e94d6c" dependencies = [ "form_urlencoded", "idna", @@ -12702,12 +12903,6 @@ version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" -[[package]] -name = "utf16_iter" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8232dd3cdaed5356e0f716d285e4b40b932ac434100fe9b7e0e8e935b9e6246" - [[package]] name = "utf8-ranges" version = "1.0.5" @@ -12720,12 +12915,6 @@ version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "86bd8d4e895da8537e5315b8254664e6b769c4ff3db18321b297a1e7004392e3" -[[package]] -name = "utf8_iter" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" - [[package]] name = "utf8parse" version = "0.2.2" @@ -12928,6 +13117,16 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + [[package]] name = "webbrowser" version = "0.8.15" @@ -13327,18 +13526,6 @@ dependencies = [ "thiserror 1.0.64", ] -[[package]] -name = "write16" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1890f4022759daae28ed4fe62859b1236caebfc61ede2f63ed4e695f3f6d936" - -[[package]] -name = "writeable" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e9df38ee2d2c3c5948ea468a8406ff0db0b29ae1ffde1bcf20ef305bcc95c51" - [[package]] name = "wyz" version = "0.5.1" @@ -13369,9 +13556,9 @@ dependencies = [ [[package]] name = "xattr" -version = "1.3.1" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8da84f1a25939b27f6820d92aed108f83ff920fdf11a7b19366c27c4cda81d4f" +checksum = "e105d177a3871454f754b33bb0ee637ecaaac997446375fd3e5d43a2ed00c909" dependencies = [ "libc", "linux-raw-sys", @@ -13402,30 +13589,6 @@ version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" -[[package]] -name = "yoke" -version = "0.7.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "120e6aef9aa629e3d4f52dc8cc43a015c7724194c97dfaf45180d2daf2b77f40" -dependencies = [ - "serde", - "stable_deref_trait", - "yoke-derive", - "zerofrom", -] - -[[package]] -name = "yoke-derive" -version = "0.7.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2380878cad4ac9aac1e2435f3eb4020e8374b5f13c296cb75b4620ff8e229154" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", - "synstructure", -] - [[package]] name = "zerocopy" version = "0.7.35" @@ -13447,27 +13610,6 @@ dependencies = [ "syn 2.0.90", ] -[[package]] -name = "zerofrom" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cff3ee08c995dee1859d998dea82f7374f2826091dd9cd47def953cae446cd2e" -dependencies = [ - "zerofrom-derive", -] - -[[package]] -name = "zerofrom-derive" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "595eed982f7d355beb85837f651fa22e90b3c044842dc7f2c2842c086f295808" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", - "synstructure", -] - [[package]] name = "zeroize" version = "1.8.1" @@ -13488,28 +13630,6 @@ dependencies = [ "syn 2.0.90", ] -[[package]] -name = "zerovec" -version = "0.10.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa2b893d79df23bfb12d5461018d408ea19dfafe76c2c7ef6d4eba614f8ff079" -dependencies = [ - "yoke", - "zerofrom", - "zerovec-derive", -] - -[[package]] -name = "zerovec-derive" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6eafa6dfb17584ea3e2bd6e76e0cc15ad7af12b09abdd1ca55961bed9b1063c6" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.90", -] - [[package]] name = "zstd" version = "0.11.2+zstd.1.5.2" diff --git a/Cargo.toml b/Cargo.toml index d582054b43..a7c5b5d5de 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -88,14 +88,17 @@ rust.unexpected_cfgs = { level = "warn", check-cfg = ['cfg(tokio_unstable)'] } # See for more detaiils: https://github.com/rust-lang/cargo/issues/11329 ahash = { version = "0.8", features = ["compile-time-rng"] } aquamarine = "0.3" -arrow = { version = "51.0.0", features = ["prettyprint"] } -arrow-array = { version = "51.0.0", default-features = false, features = ["chrono-tz"] } -arrow-flight = "51.0" -arrow-ipc = { version = "51.0.0", default-features = false, features = ["lz4", "zstd"] } -arrow-schema = { version = "51.0", features = ["serde"] } +arrow = { version = "53.0.0", features = ["prettyprint"] } +arrow-array = { version = "53.0.0", default-features = false, features = ["chrono-tz"] } +arrow-flight = "53.0" +arrow-ipc = { version = "53.0.0", default-features = false, features = ["lz4", "zstd"] } +arrow-schema = { version = "53.0", features = ["serde"] } async-stream = "0.3" async-trait = "0.1" -axum = { version = "0.6", features = ["headers"] } +# Remember to update axum-extra, axum-macros when updating axum +axum = "0.8" +axum-extra = "0.10" +axum-macros = "0.4" backon = "1" base64 = "0.21" bigdecimal = "0.4.2" @@ -107,32 +110,35 @@ clap = { version = "4.4", features = ["derive"] } config = "0.13.0" crossbeam-utils = "0.8" dashmap = "5.4" -datafusion = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-common = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-expr = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-functions = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-optimizer = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-physical-expr = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-physical-plan = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-sql = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } -datafusion-substrait = { git = "https://github.com/waynexia/arrow-datafusion.git", rev = "7823ef2f63663907edab46af0d51359900f608d6" } +datafusion = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-common = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-expr = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-functions = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-optimizer = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-physical-expr = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-physical-plan = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-sql = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } +datafusion-substrait = { git = "https://github.com/apache/datafusion.git", rev = "2464703c84c400a09cc59277018813f0e797bb4e" } deadpool = "0.10" deadpool-postgres = "0.12" derive_builder = "0.12" dotenv = "0.15" -etcd-client = "0.13" +etcd-client = "0.14" fst = "0.4.7" futures = "0.3" futures-util = "0.3" -greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "6cee3db98a552f1dd848dec3eefcce8f26343748" } +greptime-proto = { git = "https://github.com/GreptimeTeam/greptime-proto.git", rev = "683e9d10ae7f3dfb8aaabd89082fc600c17e3795" } hex = "0.4" -http = "0.2" +http = "1" humantime = "2.1" humantime-serde = "1.1" +hyper = "1.1" +hyper-util = "0.1" itertools = "0.10" jsonb = { git = "https://github.com/databendlabs/jsonb.git", rev = "8c8d2fc294a39f3ff08909d60f718639cfba3875", default-features = false } lazy_static = "1.4" local-ip-address = "0.6" +loki-api = { git = "https://github.com/shuiyisong/tracing-loki", branch = "chore/prost_version" } meter-core = { git = "https://github.com/GreptimeTeam/greptime-meter.git", rev = "a10facb353b41460eeb98578868ebf19c2084fac" } mockall = "0.11.4" moka = "0.12" @@ -140,7 +146,7 @@ nalgebra = "0.33" notify = "6.1" num_cpus = "1.16" once_cell = "1.18" -opentelemetry-proto = { version = "0.5", features = [ +opentelemetry-proto = { version = "0.27", features = [ "gen-tonic", "metrics", "trace", @@ -148,12 +154,12 @@ opentelemetry-proto = { version = "0.5", features = [ "logs", ] } parking_lot = "0.12" -parquet = { version = "51.0.0", default-features = false, features = ["arrow", "async", "object_store"] } +parquet = { version = "53.0.0", default-features = false, features = ["arrow", "async", "object_store"] } paste = "1.0" pin-project = "1.0" prometheus = { version = "0.13.3", features = ["process"] } promql-parser = { version = "0.4.3", features = ["ser"] } -prost = "0.12" +prost = "0.13" raft-engine = { version = "0.4.1", default-features = false } rand = "0.8" ratelimit = "0.9" @@ -172,6 +178,7 @@ rstest = "0.21" rstest_reuse = "0.7" rust_decimal = "1.33" rustc-hash = "2.0" +rustls = { version = "0.23.20", default-features = false } # override by patch, see [patch.crates-io] serde = { version = "1.0", features = ["derive"] } serde_json = { version = "1.0", features = ["float_roundtrip"] } serde_with = "3" @@ -180,9 +187,8 @@ similar-asserts = "1.6.0" smallvec = { version = "1", features = ["serde"] } snafu = "0.8" sysinfo = "0.30" - -rustls = { version = "0.23.20", default-features = false } # override by patch, see [patch.crates-io] -sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "54a267ac89c09b11c0c88934690530807185d3e7", features = [ +# on branch v0.52.x +sqlparser = { git = "https://github.com/GreptimeTeam/sqlparser-rs.git", rev = "71dd86058d2af97b9925093d40c4e03360403170", features = [ "visitor", "serde", ] } # on branch v0.44.x @@ -194,8 +200,8 @@ tokio-rustls = { version = "0.26.0", default-features = false } # override by pa tokio-stream = "0.1" tokio-util = { version = "0.7", features = ["io-util", "compat"] } toml = "0.8.8" -tonic = { version = "0.11", features = ["tls", "gzip", "zstd"] } -tower = "0.4" +tonic = { version = "0.12", features = ["tls", "gzip", "zstd"] } +tower = "0.5" tracing-appender = "0.2" tracing-subscriber = { version = "0.3", features = ["env-filter", "json", "fmt"] } typetag = "0.2" diff --git a/src/api/src/error.rs b/src/api/src/error.rs index 07e43e4772..01a3197bc3 100644 --- a/src/api/src/error.rs +++ b/src/api/src/error.rs @@ -33,7 +33,7 @@ pub enum Error { #[snafu(implicit)] location: Location, #[snafu(source)] - error: prost::DecodeError, + error: prost::UnknownEnumValue, }, #[snafu(display("Failed to create column datatype from {:?}", from))] diff --git a/src/api/src/helper.rs b/src/api/src/helper.rs index 59068fba76..8fd31cd241 100644 --- a/src/api/src/helper.rs +++ b/src/api/src/helper.rs @@ -86,7 +86,7 @@ impl ColumnDataTypeWrapper { /// Get a tuple of ColumnDataType and ColumnDataTypeExtension. pub fn to_parts(&self) -> (ColumnDataType, Option) { - (self.datatype, self.datatype_ext.clone()) + (self.datatype, self.datatype_ext) } } @@ -685,14 +685,18 @@ pub fn pb_values_to_vector_ref(data_type: &ConcreteDataType, values: Values) -> IntervalType::YearMonth(_) => Arc::new(IntervalYearMonthVector::from_vec( values.interval_year_month_values, )), - IntervalType::DayTime(_) => Arc::new(IntervalDayTimeVector::from_vec( - values.interval_day_time_values, + IntervalType::DayTime(_) => Arc::new(IntervalDayTimeVector::from_iter_values( + values + .interval_day_time_values + .iter() + .map(|x| IntervalDayTime::from_i64(*x).into()), )), IntervalType::MonthDayNano(_) => { Arc::new(IntervalMonthDayNanoVector::from_iter_values( - values.interval_month_day_nano_values.iter().map(|x| { - IntervalMonthDayNano::new(x.months, x.days, x.nanoseconds).to_i128() - }), + values + .interval_month_day_nano_values + .iter() + .map(|x| IntervalMonthDayNano::new(x.months, x.days, x.nanoseconds).into()), )) } }, @@ -1495,14 +1499,22 @@ mod tests { column.values.as_ref().unwrap().interval_year_month_values ); - let vector = Arc::new(IntervalDayTimeVector::from_vec(vec![4, 5, 6])); + let vector = Arc::new(IntervalDayTimeVector::from_vec(vec![ + IntervalDayTime::new(0, 4).into(), + IntervalDayTime::new(0, 5).into(), + IntervalDayTime::new(0, 6).into(), + ])); push_vals(&mut column, 3, vector); assert_eq!( vec![4, 5, 6], column.values.as_ref().unwrap().interval_day_time_values ); - let vector = Arc::new(IntervalMonthDayNanoVector::from_vec(vec![7, 8, 9])); + let vector = Arc::new(IntervalMonthDayNanoVector::from_vec(vec![ + IntervalMonthDayNano::new(0, 0, 7).into(), + IntervalMonthDayNano::new(0, 0, 8).into(), + IntervalMonthDayNano::new(0, 0, 9).into(), + ])); let len = vector.len(); push_vals(&mut column, 3, vector); (0..len).for_each(|i| { diff --git a/src/api/src/v1/column_def.rs b/src/api/src/v1/column_def.rs index 08ba010d57..6ae453cfdd 100644 --- a/src/api/src/v1/column_def.rs +++ b/src/api/src/v1/column_def.rs @@ -34,10 +34,8 @@ const SKIPPING_INDEX_GRPC_KEY: &str = "skipping_index"; /// Tries to construct a `ColumnSchema` from the given `ColumnDef`. pub fn try_as_column_schema(column_def: &ColumnDef) -> Result { - let data_type = ColumnDataTypeWrapper::try_new( - column_def.data_type, - column_def.datatype_extension.clone(), - )?; + let data_type = + ColumnDataTypeWrapper::try_new(column_def.data_type, column_def.datatype_extension)?; let constraint = if column_def.default_constraint.is_empty() { None diff --git a/src/catalog/src/lib.rs b/src/catalog/src/lib.rs index 623f2a363e..729ea58724 100644 --- a/src/catalog/src/lib.rs +++ b/src/catalog/src/lib.rs @@ -41,6 +41,7 @@ pub mod information_schema { } pub mod table_source; + #[async_trait::async_trait] pub trait CatalogManager: Send + Sync { fn as_any(&self) -> &dyn Any; diff --git a/src/catalog/src/system_schema/information_schema/cluster_info.rs b/src/catalog/src/system_schema/information_schema/cluster_info.rs index 1ab700497c..bf172b3cf0 100644 --- a/src/catalog/src/system_schema/information_schema/cluster_info.rs +++ b/src/catalog/src/system_schema/information_schema/cluster_info.rs @@ -64,6 +64,7 @@ const INIT_CAPACITY: usize = 42; /// - `uptime`: the uptime of the peer. /// - `active_time`: the time since the last activity of the peer. /// +#[derive(Debug)] pub(super) struct InformationSchemaClusterInfo { schema: SchemaRef, catalog_manager: Weak, diff --git a/src/catalog/src/system_schema/information_schema/columns.rs b/src/catalog/src/system_schema/information_schema/columns.rs index 152fc33a04..53ee5d432c 100644 --- a/src/catalog/src/system_schema/information_schema/columns.rs +++ b/src/catalog/src/system_schema/information_schema/columns.rs @@ -45,6 +45,7 @@ use crate::error::{ use crate::information_schema::Predicates; use crate::CatalogManager; +#[derive(Debug)] pub(super) struct InformationSchemaColumns { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/flows.rs b/src/catalog/src/system_schema/information_schema/flows.rs index 5d35cfbbe4..ef204e81ad 100644 --- a/src/catalog/src/system_schema/information_schema/flows.rs +++ b/src/catalog/src/system_schema/information_schema/flows.rs @@ -61,7 +61,7 @@ pub const FLOWNODE_IDS: &str = "flownode_ids"; pub const OPTIONS: &str = "options"; /// The `information_schema.flows` to provides information about flows in databases. -/// +#[derive(Debug)] pub(super) struct InformationSchemaFlows { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/key_column_usage.rs b/src/catalog/src/system_schema/information_schema/key_column_usage.rs index c88d74de0b..bd7951777d 100644 --- a/src/catalog/src/system_schema/information_schema/key_column_usage.rs +++ b/src/catalog/src/system_schema/information_schema/key_column_usage.rs @@ -62,6 +62,7 @@ pub(crate) const FULLTEXT_INDEX_CONSTRAINT_NAME: &str = "FULLTEXT INDEX"; pub(crate) const SKIPPING_INDEX_CONSTRAINT_NAME: &str = "SKIPPING INDEX"; /// The virtual table implementation for `information_schema.KEY_COLUMN_USAGE`. +#[derive(Debug)] pub(super) struct InformationSchemaKeyColumnUsage { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/partitions.rs b/src/catalog/src/system_schema/information_schema/partitions.rs index 4cfeece626..38812c2cf4 100644 --- a/src/catalog/src/system_schema/information_schema/partitions.rs +++ b/src/catalog/src/system_schema/information_schema/partitions.rs @@ -59,6 +59,7 @@ const INIT_CAPACITY: usize = 42; /// The `PARTITIONS` table provides information about partitioned tables. /// See https://dev.mysql.com/doc/refman/8.0/en/information-schema-partitions-table.html /// We provide an extral column `greptime_partition_id` for GreptimeDB region id. +#[derive(Debug)] pub(super) struct InformationSchemaPartitions { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/procedure_info.rs b/src/catalog/src/system_schema/information_schema/procedure_info.rs index 6e3c0b1f46..8dd2f32f71 100644 --- a/src/catalog/src/system_schema/information_schema/procedure_info.rs +++ b/src/catalog/src/system_schema/information_schema/procedure_info.rs @@ -56,7 +56,7 @@ const INIT_CAPACITY: usize = 42; /// - `end_time`: the ending execution time of the procedure. /// - `status`: the status of the procedure. /// - `lock_keys`: the lock keys of the procedure. -/// +#[derive(Debug)] pub(super) struct InformationSchemaProcedureInfo { schema: SchemaRef, catalog_manager: Weak, diff --git a/src/catalog/src/system_schema/information_schema/region_peers.rs b/src/catalog/src/system_schema/information_schema/region_peers.rs index 50c2593f86..2d2a42e920 100644 --- a/src/catalog/src/system_schema/information_schema/region_peers.rs +++ b/src/catalog/src/system_schema/information_schema/region_peers.rs @@ -59,7 +59,7 @@ const INIT_CAPACITY: usize = 42; /// - `is_leader`: whether the peer is the leader /// - `status`: the region status, `ALIVE` or `DOWNGRADED`. /// - `down_seconds`: the duration of being offline, in seconds. -/// +#[derive(Debug)] pub(super) struct InformationSchemaRegionPeers { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/region_statistics.rs b/src/catalog/src/system_schema/information_schema/region_statistics.rs index a98a70cb23..fdd9e59752 100644 --- a/src/catalog/src/system_schema/information_schema/region_statistics.rs +++ b/src/catalog/src/system_schema/information_schema/region_statistics.rs @@ -63,7 +63,7 @@ const INIT_CAPACITY: usize = 42; /// - `index_size`: The sst index files size in bytes. /// - `engine`: The engine type. /// - `region_role`: The region role. -/// +#[derive(Debug)] pub(super) struct InformationSchemaRegionStatistics { schema: SchemaRef, catalog_manager: Weak, diff --git a/src/catalog/src/system_schema/information_schema/runtime_metrics.rs b/src/catalog/src/system_schema/information_schema/runtime_metrics.rs index 0e519c3407..d2731b79b5 100644 --- a/src/catalog/src/system_schema/information_schema/runtime_metrics.rs +++ b/src/catalog/src/system_schema/information_schema/runtime_metrics.rs @@ -38,6 +38,7 @@ use store_api::storage::{ScanRequest, TableId}; use super::{InformationTable, RUNTIME_METRICS}; use crate::error::{CreateRecordBatchSnafu, InternalSnafu, Result}; +#[derive(Debug)] pub(super) struct InformationSchemaMetrics { schema: SchemaRef, } diff --git a/src/catalog/src/system_schema/information_schema/schemata.rs b/src/catalog/src/system_schema/information_schema/schemata.rs index d13e958131..bad9eb5f2b 100644 --- a/src/catalog/src/system_schema/information_schema/schemata.rs +++ b/src/catalog/src/system_schema/information_schema/schemata.rs @@ -49,6 +49,7 @@ pub const SCHEMA_OPTS: &str = "options"; const INIT_CAPACITY: usize = 42; /// The `information_schema.schemata` table implementation. +#[derive(Debug)] pub(super) struct InformationSchemaSchemata { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/table_constraints.rs b/src/catalog/src/system_schema/information_schema/table_constraints.rs index 50e2469946..a0be49d575 100644 --- a/src/catalog/src/system_schema/information_schema/table_constraints.rs +++ b/src/catalog/src/system_schema/information_schema/table_constraints.rs @@ -43,6 +43,7 @@ use crate::information_schema::Predicates; use crate::CatalogManager; /// The `TABLE_CONSTRAINTS` table describes which tables have constraints. +#[derive(Debug)] pub(super) struct InformationSchemaTableConstraints { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/tables.rs b/src/catalog/src/system_schema/information_schema/tables.rs index 4e56face5d..a06eb7ea7d 100644 --- a/src/catalog/src/system_schema/information_schema/tables.rs +++ b/src/catalog/src/system_schema/information_schema/tables.rs @@ -71,6 +71,7 @@ const TABLE_ID: &str = "table_id"; pub const ENGINE: &str = "engine"; const INIT_CAPACITY: usize = 42; +#[derive(Debug)] pub(super) struct InformationSchemaTables { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/information_schema/views.rs b/src/catalog/src/system_schema/information_schema/views.rs index 082e6c2ff5..0f913799c7 100644 --- a/src/catalog/src/system_schema/information_schema/views.rs +++ b/src/catalog/src/system_schema/information_schema/views.rs @@ -54,6 +54,7 @@ pub const CHARACTER_SET_CLIENT: &str = "character_set_client"; pub const COLLATION_CONNECTION: &str = "collation_connection"; /// The `information_schema.views` to provides information about views in databases. +#[derive(Debug)] pub(super) struct InformationSchemaViews { schema: SchemaRef, catalog_name: String, diff --git a/src/catalog/src/system_schema/memory_table.rs b/src/catalog/src/system_schema/memory_table.rs index f5c675832e..8d1c56131e 100644 --- a/src/catalog/src/system_schema/memory_table.rs +++ b/src/catalog/src/system_schema/memory_table.rs @@ -33,6 +33,7 @@ use super::SystemTable; use crate::error::{CreateRecordBatchSnafu, InternalSnafu, Result}; /// A memory table with specified schema and columns. +#[derive(Debug)] pub(crate) struct MemoryTable { pub(crate) table_id: TableId, pub(crate) table_name: &'static str, diff --git a/src/catalog/src/system_schema/pg_catalog/pg_class.rs b/src/catalog/src/system_schema/pg_catalog/pg_class.rs index 30476cc253..80e69eccee 100644 --- a/src/catalog/src/system_schema/pg_catalog/pg_class.rs +++ b/src/catalog/src/system_schema/pg_catalog/pg_class.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use std::fmt; use std::sync::{Arc, Weak}; use arrow_schema::SchemaRef as ArrowSchemaRef; @@ -100,6 +101,15 @@ impl PGClass { } } +impl fmt::Debug for PGClass { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PGClass") + .field("schema", &self.schema) + .field("catalog_name", &self.catalog_name) + .finish() + } +} + impl SystemTable for PGClass { fn table_id(&self) -> table::metadata::TableId { PG_CATALOG_PG_CLASS_TABLE_ID diff --git a/src/catalog/src/system_schema/pg_catalog/pg_database.rs b/src/catalog/src/system_schema/pg_catalog/pg_database.rs index 244be630fd..1881c47ff3 100644 --- a/src/catalog/src/system_schema/pg_catalog/pg_database.rs +++ b/src/catalog/src/system_schema/pg_catalog/pg_database.rs @@ -55,6 +55,15 @@ pub(super) struct PGDatabase { namespace_oid_map: PGNamespaceOidMapRef, } +impl std::fmt::Debug for PGDatabase { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PGDatabase") + .field("schema", &self.schema) + .field("catalog_name", &self.catalog_name) + .finish() + } +} + impl PGDatabase { pub(super) fn new( catalog_name: String, diff --git a/src/catalog/src/system_schema/pg_catalog/pg_namespace.rs b/src/catalog/src/system_schema/pg_catalog/pg_namespace.rs index 108cd92242..b485447ac1 100644 --- a/src/catalog/src/system_schema/pg_catalog/pg_namespace.rs +++ b/src/catalog/src/system_schema/pg_catalog/pg_namespace.rs @@ -17,6 +17,7 @@ pub(super) mod oid_map; +use std::fmt; use std::sync::{Arc, Weak}; use arrow_schema::SchemaRef as ArrowSchemaRef; @@ -87,6 +88,15 @@ impl PGNamespace { } } +impl fmt::Debug for PGNamespace { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PGNamespace") + .field("schema", &self.schema) + .field("catalog_name", &self.catalog_name) + .finish() + } +} + impl SystemTable for PGNamespace { fn schema(&self) -> SchemaRef { self.schema.clone() diff --git a/src/catalog/src/table_source.rs b/src/catalog/src/table_source.rs index ca9c5b89d3..3cb3b5087d 100644 --- a/src/catalog/src/table_source.rs +++ b/src/catalog/src/table_source.rs @@ -365,7 +365,7 @@ mod tests { Projection: person.id AS a, person.name AS b Filter: person.id > Int32(500) TableScan: person"#, - format!("\n{:?}", source.get_logical_plan().unwrap()) + format!("\n{}", source.get_logical_plan().unwrap()) ); } } diff --git a/src/catalog/src/table_source/dummy_catalog.rs b/src/catalog/src/table_source/dummy_catalog.rs index 09a703e4dd..376f77a861 100644 --- a/src/catalog/src/table_source/dummy_catalog.rs +++ b/src/catalog/src/table_source/dummy_catalog.rs @@ -15,12 +15,12 @@ //! Dummy catalog for region server. use std::any::Any; +use std::fmt; use std::sync::Arc; use async_trait::async_trait; use common_catalog::format_full_table_name; -use datafusion::catalog::schema::SchemaProvider; -use datafusion::catalog::{CatalogProvider, CatalogProviderList}; +use datafusion::catalog::{CatalogProvider, CatalogProviderList, SchemaProvider}; use datafusion::datasource::TableProvider; use snafu::OptionExt; use table::table::adapter::DfTableProviderAdapter; @@ -41,6 +41,12 @@ impl DummyCatalogList { } } +impl fmt::Debug for DummyCatalogList { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("DummyCatalogList").finish() + } +} + impl CatalogProviderList for DummyCatalogList { fn as_any(&self) -> &dyn Any { self @@ -91,6 +97,14 @@ impl CatalogProvider for DummyCatalogProvider { } } +impl fmt::Debug for DummyCatalogProvider { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("DummyCatalogProvider") + .field("catalog_name", &self.catalog_name) + .finish() + } +} + /// A dummy schema provider for [DummyCatalogList]. #[derive(Clone)] struct DummySchemaProvider { @@ -127,3 +141,12 @@ impl SchemaProvider for DummySchemaProvider { true } } + +impl fmt::Debug for DummySchemaProvider { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("DummySchemaProvider") + .field("catalog_name", &self.catalog_name) + .field("schema_name", &self.schema_name) + .finish() + } +} diff --git a/src/common/datasource/Cargo.toml b/src/common/datasource/Cargo.toml index 16137e6b3e..ae4fc221a9 100644 --- a/src/common/datasource/Cargo.toml +++ b/src/common/datasource/Cargo.toml @@ -31,7 +31,7 @@ derive_builder.workspace = true futures.workspace = true lazy_static.workspace = true object-store.workspace = true -orc-rust = { git = "https://github.com/datafusion-contrib/datafusion-orc.git", rev = "502217315726314c4008808fe169764529640599", default-features = false, features = [ +orc-rust = { version = "0.5", default-features = false, features = [ "async", ] } parquet.workspace = true diff --git a/src/common/datasource/src/file_format.rs b/src/common/datasource/src/file_format.rs index 9998303542..1991ff825e 100644 --- a/src/common/datasource/src/file_format.rs +++ b/src/common/datasource/src/file_format.rs @@ -126,8 +126,7 @@ impl ArrowDecoder for arrow::csv::reader::Decoder { } } -#[allow(deprecated)] -impl ArrowDecoder for arrow::json::RawDecoder { +impl ArrowDecoder for arrow::json::reader::Decoder { fn decode(&mut self, buf: &[u8]) -> result::Result { self.decode(buf) } diff --git a/src/common/datasource/src/file_format/csv.rs b/src/common/datasource/src/file_format/csv.rs index 66a2fc3c3f..85399684cf 100644 --- a/src/common/datasource/src/file_format/csv.rs +++ b/src/common/datasource/src/file_format/csv.rs @@ -17,8 +17,7 @@ use std::str::FromStr; use std::sync::Arc; use arrow::csv; -#[allow(deprecated)] -use arrow::csv::reader::infer_reader_schema as infer_csv_schema; +use arrow::csv::reader::Format; use arrow::record_batch::RecordBatch; use arrow_schema::{Schema, SchemaRef}; use async_trait::async_trait; @@ -161,7 +160,6 @@ impl FileOpener for CsvOpener { } } -#[allow(deprecated)] #[async_trait] impl FileFormat for CsvFormat { async fn infer_schema(&self, store: &ObjectStore, path: &str) -> Result { @@ -188,9 +186,12 @@ impl FileFormat for CsvFormat { common_runtime::spawn_blocking_global(move || { let reader = SyncIoBridge::new(decoded); - let (schema, _records_read) = - infer_csv_schema(reader, delimiter, schema_infer_max_record, has_header) - .context(error::InferSchemaSnafu)?; + let format = Format::default() + .with_delimiter(delimiter) + .with_header(has_header); + let (schema, _records_read) = format + .infer_schema(reader, schema_infer_max_record) + .context(error::InferSchemaSnafu)?; Ok(schema) }) .await @@ -253,7 +254,7 @@ mod tests { "c7: Int64: NULL", "c8: Int64: NULL", "c9: Int64: NULL", - "c10: Int64: NULL", + "c10: Utf8: NULL", "c11: Float64: NULL", "c12: Float64: NULL", "c13: Utf8: NULL" diff --git a/src/common/datasource/src/file_format/json.rs b/src/common/datasource/src/file_format/json.rs index c70a9beebb..464b4b033e 100644 --- a/src/common/datasource/src/file_format/json.rs +++ b/src/common/datasource/src/file_format/json.rs @@ -20,8 +20,7 @@ use std::sync::Arc; use arrow::datatypes::SchemaRef; use arrow::json::reader::{infer_json_schema_from_iterator, ValueIter}; use arrow::json::writer::LineDelimited; -#[allow(deprecated)] -use arrow::json::{self, RawReaderBuilder}; +use arrow::json::{self, ReaderBuilder}; use arrow::record_batch::RecordBatch; use arrow_schema::Schema; use async_trait::async_trait; @@ -140,7 +139,6 @@ impl JsonOpener { } } -#[allow(deprecated)] impl FileOpener for JsonOpener { fn open(&self, meta: FileMeta) -> DataFusionResult { open_with_decoder( @@ -148,7 +146,7 @@ impl FileOpener for JsonOpener { meta.location().to_string(), self.compression_type, || { - RawReaderBuilder::new(self.projected_schema.clone()) + ReaderBuilder::new(self.projected_schema.clone()) .with_batch_size(self.batch_size) .build_decoder() .map_err(DataFusionError::from) diff --git a/src/common/function/src/scalars/date/date_add.rs b/src/common/function/src/scalars/date/date_add.rs index 2307d2caab..b2e5e4abe9 100644 --- a/src/common/function/src/scalars/date/date_add.rs +++ b/src/common/function/src/scalars/date/date_add.rs @@ -91,6 +91,7 @@ mod tests { use std::sync::Arc; use common_query::prelude::{TypeSignature, Volatility}; + use datatypes::arrow::datatypes::IntervalDayTime; use datatypes::prelude::ConcreteDataType; use datatypes::value::Value; use datatypes::vectors::{ @@ -134,7 +135,12 @@ mod tests { let times = vec![Some(123), None, Some(42), None]; // Intervals in milliseconds - let intervals = vec![1000, 2000, 3000, 1000]; + let intervals = vec![ + IntervalDayTime::new(0, 1000), + IntervalDayTime::new(0, 2000), + IntervalDayTime::new(0, 3000), + IntervalDayTime::new(0, 1000), + ]; let results = [Some(124), None, Some(45), None]; let time_vector = TimestampSecondVector::from(times.clone()); diff --git a/src/common/function/src/scalars/date/date_sub.rs b/src/common/function/src/scalars/date/date_sub.rs index 1e00db5331..1765e5b24a 100644 --- a/src/common/function/src/scalars/date/date_sub.rs +++ b/src/common/function/src/scalars/date/date_sub.rs @@ -91,6 +91,7 @@ mod tests { use std::sync::Arc; use common_query::prelude::{TypeSignature, Volatility}; + use datatypes::arrow::datatypes::IntervalDayTime; use datatypes::prelude::ConcreteDataType; use datatypes::value::Value; use datatypes::vectors::{ @@ -139,7 +140,12 @@ mod tests { let times = vec![Some(123), None, Some(42), None]; // Intervals in milliseconds - let intervals = vec![1000, 2000, 3000, 1000]; + let intervals = vec![ + IntervalDayTime::new(0, 1000), + IntervalDayTime::new(0, 2000), + IntervalDayTime::new(0, 3000), + IntervalDayTime::new(0, 1000), + ]; let results = [Some(122), None, Some(39), None]; let time_vector = TimestampSecondVector::from(times.clone()); diff --git a/src/common/function/src/scalars/matches.rs b/src/common/function/src/scalars/matches.rs index 1b515046ab..c41b394636 100644 --- a/src/common/function/src/scalars/matches.rs +++ b/src/common/function/src/scalars/matches.rs @@ -21,10 +21,9 @@ use common_query::error::{ }; use datafusion::common::tree_node::{Transformed, TreeNode, TreeNodeIterator, TreeNodeRecursion}; use datafusion::common::{DFSchema, Result as DfResult}; -use datafusion::execution::context::SessionState; +use datafusion::execution::SessionStateBuilder; use datafusion::logical_expr::{self, Expr, Volatility}; use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner}; -use datafusion::prelude::SessionConfig; use datatypes::arrow::array::RecordBatch; use datatypes::arrow::datatypes::{DataType, Field}; use datatypes::prelude::VectorRef; @@ -104,8 +103,7 @@ impl MatchesFunction { let like_expr = ast.into_like_expr(col_name); let input_schema = Self::input_schema(); - let session_state = - SessionState::new_with_config_rt(SessionConfig::default(), Arc::default()); + let session_state = SessionStateBuilder::new().with_default_features().build(); let planner = DefaultPhysicalPlanner::default(); let physical_expr = planner .create_physical_expr(&like_expr, &input_schema, &session_state) @@ -131,7 +129,7 @@ impl MatchesFunction { } fn input_schema() -> DFSchema { - DFSchema::from_unqualifed_fields( + DFSchema::from_unqualified_fields( [Arc::new(Field::new("data", DataType::Utf8, true))].into(), HashMap::new(), ) diff --git a/src/common/function/src/system/build.rs b/src/common/function/src/system/build.rs index 925b262bcd..bd5b044a9c 100644 --- a/src/common/function/src/system/build.rs +++ b/src/common/function/src/system/build.rs @@ -42,7 +42,7 @@ impl Function for BuildFunction { } fn signature(&self) -> Signature { - Signature::uniform(0, vec![], Volatility::Immutable) + Signature::nullary(Volatility::Immutable) } fn eval(&self, _func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result { @@ -56,8 +56,6 @@ impl Function for BuildFunction { mod tests { use std::sync::Arc; - use common_query::prelude::TypeSignature; - use super::*; #[test] fn test_build_function() { @@ -67,12 +65,7 @@ mod tests { ConcreteDataType::string_datatype(), build.return_type(&[]).unwrap() ); - assert!(matches!(build.signature(), - Signature { - type_signature: TypeSignature::Uniform(0, valid_types), - volatility: Volatility::Immutable - } if valid_types.is_empty() - )); + assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable)); let build_info = common_version::build_info().to_string(); let vector = build.eval(FunctionContext::default(), &[]).unwrap(); let expect: VectorRef = Arc::new(StringVector::from(vec![build_info])); diff --git a/src/common/function/src/system/database.rs b/src/common/function/src/system/database.rs index ae0985a084..a9759de115 100644 --- a/src/common/function/src/system/database.rs +++ b/src/common/function/src/system/database.rs @@ -44,7 +44,7 @@ impl Function for DatabaseFunction { } fn signature(&self) -> Signature { - Signature::uniform(0, vec![], Volatility::Immutable) + Signature::nullary(Volatility::Immutable) } fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result { @@ -116,7 +116,6 @@ impl fmt::Display for SessionUserFunction { mod tests { use std::sync::Arc; - use common_query::prelude::TypeSignature; use session::context::QueryContextBuilder; use super::*; @@ -128,12 +127,7 @@ mod tests { ConcreteDataType::string_datatype(), build.return_type(&[]).unwrap() ); - assert!(matches!(build.signature(), - Signature { - type_signature: TypeSignature::Uniform(0, valid_types), - volatility: Volatility::Immutable - } if valid_types == vec![] - )); + assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable)); let query_ctx = QueryContextBuilder::default() .current_schema("test_db".to_string()) diff --git a/src/common/function/src/system/timezone.rs b/src/common/function/src/system/timezone.rs index f9c824ab33..1ac873e61b 100644 --- a/src/common/function/src/system/timezone.rs +++ b/src/common/function/src/system/timezone.rs @@ -38,7 +38,7 @@ impl Function for TimezoneFunction { } fn signature(&self) -> Signature { - Signature::uniform(0, vec![], Volatility::Immutable) + Signature::nullary(Volatility::Immutable) } fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result { @@ -58,7 +58,6 @@ impl fmt::Display for TimezoneFunction { mod tests { use std::sync::Arc; - use common_query::prelude::TypeSignature; use session::context::QueryContextBuilder; use super::*; @@ -70,12 +69,7 @@ mod tests { ConcreteDataType::string_datatype(), build.return_type(&[]).unwrap() ); - assert!(matches!(build.signature(), - Signature { - type_signature: TypeSignature::Uniform(0, valid_types), - volatility: Volatility::Immutable - } if valid_types == vec![] - )); + assert_eq!(build.signature(), Signature::nullary(Volatility::Immutable)); let query_ctx = QueryContextBuilder::default().build().into(); diff --git a/src/common/function/src/system/version.rs b/src/common/function/src/system/version.rs index 0843ec2363..96a8d7fc6b 100644 --- a/src/common/function/src/system/version.rs +++ b/src/common/function/src/system/version.rs @@ -42,7 +42,7 @@ impl Function for VersionFunction { } fn signature(&self) -> Signature { - Signature::exact(vec![], Volatility::Immutable) + Signature::nullary(Volatility::Immutable) } fn eval(&self, func_ctx: FunctionContext, _columns: &[VectorRef]) -> Result { diff --git a/src/common/grpc-expr/src/error.rs b/src/common/grpc-expr/src/error.rs index 171b10254d..7eca56bccd 100644 --- a/src/common/grpc-expr/src/error.rs +++ b/src/common/grpc-expr/src/error.rs @@ -107,7 +107,7 @@ pub enum Error { #[snafu(implicit)] location: Location, #[snafu(source)] - error: prost::DecodeError, + error: prost::UnknownEnumValue, }, #[snafu(display( @@ -137,7 +137,7 @@ pub enum Error { #[snafu(implicit)] location: Location, #[snafu(source)] - error: prost::DecodeError, + error: prost::UnknownEnumValue, }, #[snafu(display("Missing alter index options"))] diff --git a/src/common/grpc-expr/src/insert.rs b/src/common/grpc-expr/src/insert.rs index 9ce9ff29cc..7852403256 100644 --- a/src/common/grpc-expr/src/insert.rs +++ b/src/common/grpc-expr/src/insert.rs @@ -255,7 +255,7 @@ mod tests { ConcreteDataType::from( ColumnDataTypeWrapper::try_new( decimal_column.data_type, - decimal_column.datatype_extension.clone(), + decimal_column.datatype_extension, ) .unwrap() ) @@ -351,7 +351,6 @@ mod tests { .as_ref() .unwrap() .datatype_extension - .clone() ) .unwrap() ) diff --git a/src/common/grpc-expr/src/util.rs b/src/common/grpc-expr/src/util.rs index b7bf50b932..9be5a410d7 100644 --- a/src/common/grpc-expr/src/util.rs +++ b/src/common/grpc-expr/src/util.rs @@ -166,7 +166,7 @@ pub fn build_create_table_expr( default_constraint: vec![], semantic_type, comment: String::new(), - datatype_extension: datatype_extension.clone(), + datatype_extension: *datatype_extension, options: options.clone(), }); } @@ -208,7 +208,7 @@ pub fn extract_new_columns( default_constraint: vec![], semantic_type: expr.semantic_type, comment: String::new(), - datatype_extension: expr.datatype_extension.clone(), + datatype_extension: *expr.datatype_extension, options: expr.options.clone(), }); AddColumn { diff --git a/src/common/grpc/Cargo.toml b/src/common/grpc/Cargo.toml index a823e427a5..d20e751e41 100644 --- a/src/common/grpc/Cargo.toml +++ b/src/common/grpc/Cargo.toml @@ -19,7 +19,8 @@ common-telemetry.workspace = true common-time.workspace = true dashmap.workspace = true datatypes.workspace = true -flatbuffers = "23.1" +flatbuffers = "24" +hyper.workspace = true lazy_static.workspace = true prost.workspace = true snafu.workspace = true @@ -29,6 +30,7 @@ tower.workspace = true [dev-dependencies] criterion = "0.4" +hyper-util = { workspace = true, features = ["tokio"] } rand.workspace = true [[bench]] diff --git a/src/common/grpc/src/channel_manager.rs b/src/common/grpc/src/channel_manager.rs index eee173c3aa..0127829567 100644 --- a/src/common/grpc/src/channel_manager.rs +++ b/src/common/grpc/src/channel_manager.rs @@ -25,7 +25,7 @@ use snafu::{OptionExt, ResultExt}; use tonic::transport::{ Certificate, Channel as InnerChannel, ClientTlsConfig, Endpoint, Identity, Uri, }; -use tower::make::MakeConnection; +use tower::Service; use crate::error::{CreateChannelSnafu, InvalidConfigFilePathSnafu, InvalidTlsConfigSnafu, Result}; @@ -137,8 +137,8 @@ impl ChannelManager { connector: C, ) -> Result where - C: MakeConnection + Send + 'static, - C::Connection: Unpin + Send + 'static, + C: Service + Send + 'static, + C::Response: hyper::rt::Read + hyper::rt::Write + Send + Unpin, C::Future: Send + 'static, Box: From + Send + 'static, { @@ -607,7 +607,7 @@ mod tests { }); let (client, _) = tokio::io::duplex(1024); - let mut client = Some(client); + let mut client = Some(hyper_util::rt::TokioIo::new(client)); let res = mgr.reset_with_connector( addr, service_fn(move |_| { diff --git a/src/common/grpc/src/select.rs b/src/common/grpc/src/select.rs index bf0dd918dd..25fad094e6 100644 --- a/src/common/grpc/src/select.rs +++ b/src/common/grpc/src/select.rs @@ -205,7 +205,7 @@ pub fn values(arrays: &[VectorRef]) -> Result { ConcreteDataType::Interval(IntervalType::DayTime(_)), IntervalDayTimeVector, interval_day_time_values, - |x| { x.into_native() } + |x| { x.to_i64() } ), ( ConcreteDataType::Interval(IntervalType::MonthDayNano(_)), @@ -232,6 +232,8 @@ pub fn values(arrays: &[VectorRef]) -> Result { mod tests { use std::sync::Arc; + use datatypes::arrow::datatypes::{IntervalDayTime, IntervalMonthDayNano}; + use super::*; #[test] @@ -266,7 +268,12 @@ mod tests { #[test] fn test_convert_arrow_array_interval_day_time() { - let array = IntervalDayTimeVector::from(vec![Some(1), Some(2), None, Some(3)]); + let array = IntervalDayTimeVector::from(vec![ + Some(IntervalDayTime::new(0, 1)), + Some(IntervalDayTime::new(0, 2)), + None, + Some(IntervalDayTime::new(0, 3)), + ]); let array: VectorRef = Arc::new(array); let values = values(&[array]).unwrap(); @@ -276,7 +283,12 @@ mod tests { #[test] fn test_convert_arrow_array_interval_month_day_nano() { - let array = IntervalMonthDayNanoVector::from(vec![Some(1), Some(2), None, Some(3)]); + let array = IntervalMonthDayNanoVector::from(vec![ + Some(IntervalMonthDayNano::new(0, 0, 1)), + Some(IntervalMonthDayNano::new(0, 0, 2)), + None, + Some(IntervalMonthDayNano::new(0, 0, 3)), + ]); let array: VectorRef = Arc::new(array); let values = values(&[array]).unwrap(); diff --git a/src/common/macro/src/range_fn.rs b/src/common/macro/src/range_fn.rs index 882de64aa4..bf415240d9 100644 --- a/src/common/macro/src/range_fn.rs +++ b/src/common/macro/src/range_fn.rs @@ -138,7 +138,7 @@ fn build_struct( datafusion_expr::create_udf( Self::name(), Self::input_type(), - Arc::new(Self::return_type()), + Self::return_type(), Volatility::Immutable, Arc::new(Self::calc) as _, ) diff --git a/src/common/meta/src/ddl/create_table_template.rs b/src/common/meta/src/ddl/create_table_template.rs index 7da347bda3..fefb47ce37 100644 --- a/src/common/meta/src/ddl/create_table_template.rs +++ b/src/common/meta/src/ddl/create_table_template.rs @@ -47,7 +47,7 @@ pub(crate) fn build_template(create_table_expr: &CreateTableExpr) -> Result) -> std::fmt::Result { + f.debug_struct("FlowMetadataManager").finish() + } +} + #[cfg(test)] mod tests { use std::assert_matches::assert_matches; diff --git a/src/common/meta/src/state_store.rs b/src/common/meta/src/state_store.rs index a9c3f14a3e..8bae361584 100644 --- a/src/common/meta/src/state_store.rs +++ b/src/common/meta/src/state_store.rs @@ -290,7 +290,8 @@ mod tests { num_per_range: u32, max_bytes: u32, ) { - let num_cases = rand::thread_rng().gen_range(1..=26); + let num_cases = rand::thread_rng().gen_range(1..=8); + common_telemetry::info!("num_cases: {}", num_cases); let mut cases = Vec::with_capacity(num_cases); for i in 0..num_cases { let size = rand::thread_rng().gen_range(size_limit..=max_bytes); @@ -324,6 +325,7 @@ mod tests { // Puts the values for TestCase { key, value, .. } in &cases { + common_telemetry::info!("put key: {}, size: {}", key, value.len()); store.put(key, value.clone()).await.unwrap(); } @@ -332,6 +334,7 @@ mod tests { let data = walk_top_down(prefix).await; assert_eq!(data.len(), 1); let (keyset, got) = data.into_iter().next().unwrap(); + common_telemetry::info!("get key: {}", keyset.key()); let num_expected_keys = value.len().div_ceil(size_limit as usize); assert_eq!(&got, value); assert_eq!(keyset.key(), key); @@ -364,6 +367,7 @@ mod tests { let prefix = "test_etcd_store_split_value/"; let endpoints = env::var("GT_ETCD_ENDPOINTS").unwrap_or_default(); let kv_backend: KvBackendRef = if endpoints.is_empty() { + common_telemetry::info!("Using MemoryKvBackend"); Arc::new(MemoryKvBackend::new()) } else { let endpoints = endpoints diff --git a/src/common/query/src/error.rs b/src/common/query/src/error.rs index 7e8d6b79dc..4141f34881 100644 --- a/src/common/query/src/error.rs +++ b/src/common/query/src/error.rs @@ -237,6 +237,15 @@ pub enum Error { #[snafu(implicit)] location: Location, }, + + #[snafu(display("Failed to register UDF: {}", name))] + RegisterUdf { + name: String, + #[snafu(source)] + error: DataFusionError, + #[snafu(implicit)] + location: Location, + }, } pub type Result = std::result::Result; @@ -268,7 +277,8 @@ impl ErrorExt for Error { Error::MissingTableMutationHandler { .. } | Error::MissingProcedureServiceHandler { .. } - | Error::MissingFlowServiceHandler { .. } => StatusCode::Unexpected, + | Error::MissingFlowServiceHandler { .. } + | Error::RegisterUdf { .. } => StatusCode::Unexpected, Error::UnsupportedInputDataType { .. } | Error::TypeCast { .. } diff --git a/src/common/query/src/logical_plan.rs b/src/common/query/src/logical_plan.rs index 7fd081c219..135ad63918 100644 --- a/src/common/query/src/logical_plan.rs +++ b/src/common/query/src/logical_plan.rs @@ -214,8 +214,7 @@ mod tests { ]; // call the function - let result = (df_udf.fun())(&args).unwrap(); - + let result = df_udf.invoke_batch(&args, 4).unwrap(); match result { DfColumnarValue::Array(arr) => { let arr = arr.as_any().downcast_ref::().unwrap(); @@ -308,7 +307,7 @@ mod tests { Projection: person.id AS a, person.name AS b Filter: person.id > Int32(500) TableScan: person"#, - format!("\n{:?}", new_plan) + format!("\n{}", new_plan) ); } diff --git a/src/common/query/src/signature.rs b/src/common/query/src/signature.rs index 13eaf49e2b..e74baeddcf 100644 --- a/src/common/query/src/signature.rs +++ b/src/common/query/src/signature.rs @@ -39,6 +39,10 @@ pub enum TypeSignature { Any(usize), /// One of a list of signatures OneOf(Vec), + /// Zero argument + /// This is the new signature for functions with zero arguments + /// TODO(discord9): make all other usize nonzero usize + NullAry, } ///The Signature of a function defines its supported input types as well as its volatility. @@ -112,6 +116,13 @@ impl Signature { volatility, } } + + pub fn nullary(volatility: Volatility) -> Self { + Signature { + type_signature: TypeSignature::NullAry, + volatility, + } + } } /// Conversations between datafusion signature and our signature @@ -122,16 +133,25 @@ impl From for DfTypeSignature { DfTypeSignature::Variadic(concrete_types_to_arrow_types(types)) } TypeSignature::Uniform(n, types) => { + if n == 0 { + return DfTypeSignature::NullAry; + } DfTypeSignature::Uniform(n, concrete_types_to_arrow_types(types)) } TypeSignature::Exact(types) => { DfTypeSignature::Exact(concrete_types_to_arrow_types(types)) } - TypeSignature::Any(n) => DfTypeSignature::Any(n), + TypeSignature::Any(n) => { + if n == 0 { + return DfTypeSignature::NullAry; + } + DfTypeSignature::Any(n) + } TypeSignature::OneOf(ts) => { DfTypeSignature::OneOf(ts.into_iter().map(Into::into).collect()) } TypeSignature::VariadicAny => DfTypeSignature::VariadicAny, + TypeSignature::NullAry => DfTypeSignature::NullAry, } } } diff --git a/src/common/query/src/stream.rs b/src/common/query/src/stream.rs index a1ed81a583..2fefb4745d 100644 --- a/src/common/query/src/stream.rs +++ b/src/common/query/src/stream.rs @@ -119,6 +119,10 @@ impl ExecutionPlan for StreamScanAdapter { .ok_or_else(|| DataFusionError::Execution("Stream already exhausted".to_string()))?; Ok(Box::pin(DfRecordBatchStreamAdapter::new(stream))) } + + fn name(&self) -> &str { + "StreamScanAdapter" + } } #[cfg(test)] diff --git a/src/common/recordbatch/src/recordbatch.rs b/src/common/recordbatch/src/recordbatch.rs index 901c5bd12b..efbf5f804b 100644 --- a/src/common/recordbatch/src/recordbatch.rs +++ b/src/common/recordbatch/src/recordbatch.rs @@ -17,6 +17,7 @@ use std::slice; use std::sync::Arc; use datafusion::arrow::util::pretty::pretty_format_batches; +use datatypes::arrow::array::RecordBatchOptions; use datatypes::prelude::DataType; use datatypes::schema::SchemaRef; use datatypes::value::Value; @@ -73,6 +74,21 @@ impl RecordBatch { } } + /// Create an empty [`RecordBatch`] from `schema` with `num_rows`. + pub fn new_with_count(schema: SchemaRef, num_rows: usize) -> Result { + let df_record_batch = DfRecordBatch::try_new_with_options( + schema.arrow_schema().clone(), + vec![], + &RecordBatchOptions::new().with_row_count(Some(num_rows)), + ) + .context(error::NewDfRecordBatchSnafu)?; + Ok(RecordBatch { + schema, + columns: vec![], + df_record_batch, + }) + } + pub fn try_project(&self, indices: &[usize]) -> Result { let schema = Arc::new(self.schema.try_project(indices).context(DataTypesSnafu)?); let mut columns = Vec::with_capacity(indices.len()); diff --git a/src/common/runtime/src/runtime.rs b/src/common/runtime/src/runtime.rs index cec201bc25..d186b9d2f0 100644 --- a/src/common/runtime/src/runtime.rs +++ b/src/common/runtime/src/runtime.rs @@ -142,16 +142,25 @@ impl Builder { impl BuilderBuild for Builder { fn build(&mut self) -> Result { - let runtime = self + let builder = self .builder .enable_all() .thread_name(self.thread_name.clone()) .on_thread_start(on_thread_start(self.thread_name.clone())) .on_thread_stop(on_thread_stop(self.thread_name.clone())) .on_thread_park(on_thread_park(self.thread_name.clone())) - .on_thread_unpark(on_thread_unpark(self.thread_name.clone())) - .build() - .context(BuildRuntimeSnafu)?; + .on_thread_unpark(on_thread_unpark(self.thread_name.clone())); + let runtime = if cfg!(debug_assertions) { + // Set the stack size to 8MB for the thread so it wouldn't overflow on large stack usage in debug mode + // This is necessary to avoid stack overflow while running sqlness. + // https://github.com/rust-lang/rust/issues/34283 + builder + .thread_stack_size(8 * 1024 * 1024) + .build() + .context(BuildRuntimeSnafu)? + } else { + builder.build().context(BuildRuntimeSnafu)? + }; let name = self.runtime_name.clone(); let handle = runtime.handle().clone(); diff --git a/src/common/substrait/src/df_substrait.rs b/src/common/substrait/src/df_substrait.rs index 9217b60cc5..a03acace4a 100644 --- a/src/common/substrait/src/df_substrait.rs +++ b/src/common/substrait/src/df_substrait.rs @@ -16,10 +16,10 @@ use std::sync::Arc; use async_trait::async_trait; use bytes::{Buf, Bytes, BytesMut}; -use datafusion::catalog::CatalogProviderList; use datafusion::execution::context::SessionState; use datafusion::execution::runtime_env::RuntimeEnv; -use datafusion::prelude::{SessionConfig, SessionContext}; +use datafusion::execution::SessionStateBuilder; +use datafusion::prelude::SessionConfig; use datafusion_expr::LogicalPlan; use datafusion_substrait::logical_plan::consumer::from_substrait_plan; use datafusion_substrait::logical_plan::producer::to_substrait_plan; @@ -41,13 +41,10 @@ impl SubstraitPlan for DFLogicalSubstraitConvertor { async fn decode( &self, message: B, - catalog_list: Arc, state: SessionState, ) -> Result { - let mut context = SessionContext::new_with_state(state); - context.register_catalog_list(catalog_list); let plan = Plan::decode(message).context(DecodeRelSnafu)?; - let df_plan = from_substrait_plan(&context, &plan) + let df_plan = from_substrait_plan(&state, &plan) .await .context(DecodeDfPlanSnafu)?; Ok(df_plan) @@ -72,11 +69,12 @@ impl DFLogicalSubstraitConvertor { plan: &LogicalPlan, serializer: impl SerializerRegistry + 'static, ) -> Result, Error> { - let session_state = - SessionState::new_with_config_rt(SessionConfig::new(), Arc::new(RuntimeEnv::default())) - .with_serializer_registry(Arc::new(serializer)); - let context = SessionContext::new_with_state(session_state); - - to_substrait_plan(plan, &context).context(EncodeDfPlanSnafu) + let state = SessionStateBuilder::new() + .with_config(SessionConfig::new()) + .with_runtime_env(Arc::new(RuntimeEnv::default())) + .with_default_features() + .with_serializer_registry(Arc::new(serializer)) + .build(); + to_substrait_plan(plan, &state).context(EncodeDfPlanSnafu) } } diff --git a/src/common/substrait/src/extension_serializer.rs b/src/common/substrait/src/extension_serializer.rs index a817943768..48d86e2c59 100644 --- a/src/common/substrait/src/extension_serializer.rs +++ b/src/common/substrait/src/extension_serializer.rs @@ -22,6 +22,7 @@ use promql::extension_plan::{ EmptyMetric, InstantManipulate, RangeManipulate, ScalarCalculate, SeriesDivide, SeriesNormalize, }; +#[derive(Debug)] pub struct ExtensionSerializer; impl SerializerRegistry for ExtensionSerializer { diff --git a/src/common/substrait/src/lib.rs b/src/common/substrait/src/lib.rs index 756e701c48..f5f4586939 100644 --- a/src/common/substrait/src/lib.rs +++ b/src/common/substrait/src/lib.rs @@ -17,11 +17,9 @@ mod df_substrait; pub mod error; pub mod extension_serializer; -use std::sync::Arc; use async_trait::async_trait; use bytes::{Buf, Bytes}; -use datafusion::catalog::CatalogProviderList; use datafusion::execution::context::SessionState; pub use datafusion::execution::registry::SerializerRegistry; /// Re-export the Substrait module of datafusion, @@ -40,7 +38,6 @@ pub trait SubstraitPlan { async fn decode( &self, message: B, - catalog_list: Arc, state: SessionState, ) -> Result; diff --git a/src/common/test-util/src/ports.rs b/src/common/test-util/src/ports.rs index 659e08b047..51818c15b5 100644 --- a/src/common/test-util/src/ports.rs +++ b/src/common/test-util/src/ports.rs @@ -22,6 +22,6 @@ static PORTS: OnceCell = OnceCell::new(); /// Return a unique port(in runtime) for test pub fn get_port() -> usize { PORTS - .get_or_init(|| AtomicUsize::new(rand::thread_rng().gen_range(3000..3800))) + .get_or_init(|| AtomicUsize::new(rand::thread_rng().gen_range(13000..13800))) .fetch_add(1, Ordering::Relaxed) } diff --git a/src/common/time/src/interval.rs b/src/common/time/src/interval.rs index 0ca40f7d79..4cb8c45c55 100644 --- a/src/common/time/src/interval.rs +++ b/src/common/time/src/interval.rs @@ -184,6 +184,24 @@ impl From for serde_json::Value { } } +impl From for IntervalDayTime { + fn from(value: arrow::datatypes::IntervalDayTime) -> Self { + Self { + days: value.days, + milliseconds: value.milliseconds, + } + } +} + +impl From for arrow::datatypes::IntervalDayTime { + fn from(value: IntervalDayTime) -> Self { + Self { + days: value.days, + milliseconds: value.milliseconds, + } + } +} + // Millisecond convert to other time unit pub const MS_PER_SEC: i64 = 1_000; pub const MS_PER_MINUTE: i64 = 60 * MS_PER_SEC; @@ -283,6 +301,26 @@ impl From for serde_json::Value { } } +impl From for IntervalMonthDayNano { + fn from(value: arrow::datatypes::IntervalMonthDayNano) -> Self { + Self { + months: value.months, + days: value.days, + nanoseconds: value.nanoseconds, + } + } +} + +impl From for arrow::datatypes::IntervalMonthDayNano { + fn from(value: IntervalMonthDayNano) -> Self { + Self { + months: value.months, + days: value.days, + nanoseconds: value.nanoseconds, + } + } +} + // Nanosecond convert to other time unit pub const NS_PER_SEC: i64 = 1_000_000_000; pub const NS_PER_MINUTE: i64 = 60 * NS_PER_SEC; diff --git a/src/datatypes/src/interval.rs b/src/datatypes/src/interval.rs index b724de8022..b15678c94f 100644 --- a/src/datatypes/src/interval.rs +++ b/src/datatypes/src/interval.rs @@ -12,6 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. +use arrow::datatypes::{ + IntervalDayTime as ArrowIntervalDayTime, IntervalMonthDayNano as ArrowIntervalMonthDayNano, +}; use common_time::{IntervalDayTime, IntervalMonthDayNano, IntervalYearMonth}; use paste::paste; @@ -53,11 +56,11 @@ macro_rules! define_interval_with_unit { type Native = $native_ty; fn from_native(value: Self::Native) -> Self { - Self::[](value) + Self::from(value) } fn into_native(self) -> Self::Native { - self.[]() + self.into() } } } @@ -65,8 +68,8 @@ macro_rules! define_interval_with_unit { } define_interval_with_unit!(YearMonth, i32); -define_interval_with_unit!(DayTime, i64); -define_interval_with_unit!(MonthDayNano, i128); +define_interval_with_unit!(DayTime, ArrowIntervalDayTime); +define_interval_with_unit!(MonthDayNano, ArrowIntervalMonthDayNano); #[cfg(test)] mod tests { @@ -82,12 +85,15 @@ mod tests { let interval = IntervalDayTime::from(1000); assert_eq!(interval, interval.as_scalar_ref()); assert_eq!(interval, interval.to_owned_scalar()); - assert_eq!(1000, interval.into_native()); + assert_eq!(ArrowIntervalDayTime::from(interval), interval.into_native()); let interval = IntervalMonthDayNano::from(1000); assert_eq!(interval, interval.as_scalar_ref()); assert_eq!(interval, interval.to_owned_scalar()); - assert_eq!(1000, interval.into_native()); + assert_eq!( + ArrowIntervalMonthDayNano::from(interval), + interval.into_native() + ); } #[test] diff --git a/src/datatypes/src/schema/column_schema.rs b/src/datatypes/src/schema/column_schema.rs index 6808bbab2f..5146b640fc 100644 --- a/src/datatypes/src/schema/column_schema.rs +++ b/src/datatypes/src/schema/column_schema.rs @@ -447,7 +447,21 @@ impl TryFrom<&Field> for ColumnSchema { } None => None, }; - let is_time_index = metadata.contains_key(TIME_INDEX_KEY); + let mut is_time_index = metadata.contains_key(TIME_INDEX_KEY); + if is_time_index && !data_type.is_timestamp() { + // If the column is time index but the data type is not timestamp, it is invalid. + // We set the time index to false and remove the metadata. + // This is possible if we cast the time index column to another type. DataFusion will + // keep the metadata: + // https://github.com/apache/datafusion/pull/12951 + is_time_index = false; + metadata.remove(TIME_INDEX_KEY); + common_telemetry::debug!( + "Column {} is not timestamp ({:?}) but has time index metadata", + data_type, + field.name(), + ); + } Ok(ColumnSchema { name: field.name().clone(), @@ -657,7 +671,7 @@ impl TryFrom> for SkippingIndexOptions { mod tests { use std::sync::Arc; - use arrow::datatypes::DataType as ArrowDataType; + use arrow::datatypes::{DataType as ArrowDataType, TimeUnit}; use super::*; use crate::value::Value; @@ -879,4 +893,40 @@ mod tests { &ConcreteDataType::vector_datatype(3).name() ); } + + #[test] + fn test_column_schema_fix_time_index() { + let field = Field::new( + "test", + ArrowDataType::Timestamp(TimeUnit::Second, None), + false, + ); + let field = field.with_metadata(Metadata::from([( + TIME_INDEX_KEY.to_string(), + "true".to_string(), + )])); + let column_schema = ColumnSchema::try_from(&field).unwrap(); + assert_eq!("test", column_schema.name); + assert_eq!( + ConcreteDataType::timestamp_second_datatype(), + column_schema.data_type + ); + assert!(!column_schema.is_nullable); + assert!(column_schema.is_time_index); + assert!(column_schema.default_constraint.is_none()); + assert_eq!(1, column_schema.metadata().len()); + + let field = Field::new("test", ArrowDataType::Int32, false); + let field = field.with_metadata(Metadata::from([( + TIME_INDEX_KEY.to_string(), + "true".to_string(), + )])); + let column_schema = ColumnSchema::try_from(&field).unwrap(); + assert_eq!("test", column_schema.name); + assert_eq!(ConcreteDataType::int32_datatype(), column_schema.data_type); + assert!(!column_schema.is_nullable); + assert!(!column_schema.is_time_index); + assert!(column_schema.default_constraint.is_none()); + assert!(column_schema.metadata.is_empty()); + } } diff --git a/src/datatypes/src/types/interval_type.rs b/src/datatypes/src/types/interval_type.rs index 77f1b47d6d..e6b6544d0d 100644 --- a/src/datatypes/src/types/interval_type.rs +++ b/src/datatypes/src/types/interval_type.rs @@ -13,7 +13,9 @@ // limitations under the License. use arrow::datatypes::{ - DataType as ArrowDataType, IntervalDayTimeType as ArrowIntervalDayTimeType, + DataType as ArrowDataType, IntervalDayTime as ArrowIntervalDayTime, + IntervalDayTimeType as ArrowIntervalDayTimeType, + IntervalMonthDayNano as ArrowIntervalMonthDayNano, IntervalMonthDayNanoType as ArrowIntervalMonthDayNanoType, IntervalUnit as ArrowIntervalUnit, IntervalYearMonthType as ArrowIntervalYearMonthType, }; @@ -137,8 +139,8 @@ macro_rules! impl_data_type_for_interval { } impl_data_type_for_interval!(YearMonth, i32); -impl_data_type_for_interval!(DayTime, i64); -impl_data_type_for_interval!(MonthDayNano, i128); +impl_data_type_for_interval!(DayTime, ArrowIntervalDayTime); +impl_data_type_for_interval!(MonthDayNano, ArrowIntervalMonthDayNano); #[cfg(test)] mod tests { diff --git a/src/datatypes/src/value.rs b/src/datatypes/src/value.rs index b57b364cf3..25aa9045e8 100644 --- a/src/datatypes/src/value.rs +++ b/src/datatypes/src/value.rs @@ -411,8 +411,8 @@ impl Value { Value::Timestamp(t) => timestamp_to_scalar_value(t.unit(), Some(t.value())), Value::Time(t) => time_to_scalar_value(*t.unit(), Some(t.value()))?, Value::IntervalYearMonth(v) => ScalarValue::IntervalYearMonth(Some(v.to_i32())), - Value::IntervalDayTime(v) => ScalarValue::IntervalDayTime(Some(v.to_i64())), - Value::IntervalMonthDayNano(v) => ScalarValue::IntervalMonthDayNano(Some(v.to_i128())), + Value::IntervalDayTime(v) => ScalarValue::IntervalDayTime(Some((*v).into())), + Value::IntervalMonthDayNano(v) => ScalarValue::IntervalMonthDayNano(Some((*v).into())), Value::Duration(d) => duration_to_scalar_value(d.unit(), Some(d.value())), Value::Decimal128(d) => { let (v, p, s) = d.to_scalar_value(); @@ -852,6 +852,7 @@ impl ListValue { Ok(ScalarValue::List(ScalarValue::new_list( &vs, &self.datatype.as_arrow_type(), + true, ))) } @@ -964,10 +965,10 @@ impl TryFrom for Value { .map(|x| Value::IntervalYearMonth(IntervalYearMonth::from_i32(x))) .unwrap_or(Value::Null), ScalarValue::IntervalDayTime(t) => t - .map(|x| Value::IntervalDayTime(IntervalDayTime::from_i64(x))) + .map(|x| Value::IntervalDayTime(IntervalDayTime::from(x))) .unwrap_or(Value::Null), ScalarValue::IntervalMonthDayNano(t) => t - .map(|x| Value::IntervalMonthDayNano(IntervalMonthDayNano::from_i128(x))) + .map(|x| Value::IntervalMonthDayNano(IntervalMonthDayNano::from(x))) .unwrap_or(Value::Null), ScalarValue::DurationSecond(d) => d .map(|x| Value::Duration(Duration::new(x, TimeUnit::Second))) @@ -990,7 +991,10 @@ impl TryFrom for Value { | ScalarValue::LargeList(_) | ScalarValue::Dictionary(_, _) | ScalarValue::Union(_, _, _) - | ScalarValue::Float16(_) => { + | ScalarValue::Float16(_) + | ScalarValue::Utf8View(_) + | ScalarValue::BinaryView(_) + | ScalarValue::Map(_) => { return error::UnsupportedArrowTypeSnafu { arrow_type: v.data_type(), } @@ -1733,6 +1737,7 @@ mod tests { ScalarValue::List(ScalarValue::new_list( &[ScalarValue::Int32(Some(1)), ScalarValue::Int32(None)], &ArrowDataType::Int32, + true, )) .try_into() .unwrap() @@ -1742,7 +1747,7 @@ mod tests { vec![], ConcreteDataType::list_datatype(ConcreteDataType::uint32_datatype()) )), - ScalarValue::List(ScalarValue::new_list(&[], &ArrowDataType::UInt32)) + ScalarValue::List(ScalarValue::new_list(&[], &ArrowDataType::UInt32, true)) .try_into() .unwrap() ); @@ -1814,7 +1819,7 @@ mod tests { ); assert_eq!( Value::IntervalMonthDayNano(IntervalMonthDayNano::new(1, 1, 1)), - ScalarValue::IntervalMonthDayNano(Some(IntervalMonthDayNano::new(1, 1, 1).to_i128())) + ScalarValue::IntervalMonthDayNano(Some(IntervalMonthDayNano::new(1, 1, 1).into())) .try_into() .unwrap() ); diff --git a/src/datatypes/src/vectors/eq.rs b/src/datatypes/src/vectors/eq.rs index ae923f58dc..974aa3dcad 100644 --- a/src/datatypes/src/vectors/eq.rs +++ b/src/datatypes/src/vectors/eq.rs @@ -159,6 +159,8 @@ fn equal(lhs: &dyn Vector, rhs: &dyn Vector) -> bool { #[cfg(test)] mod tests { + use arrow::datatypes::{IntervalDayTime, IntervalMonthDayNano}; + use super::*; use crate::vectors::{ list, DurationMicrosecondVector, DurationMillisecondVector, DurationNanosecondVector, @@ -236,10 +238,16 @@ mod tests { 1000, 2000, 3000, 4000, ]))); assert_vector_ref_eq(Arc::new(IntervalDayTimeVector::from_values([ - 1000, 2000, 3000, 4000, + IntervalDayTime::new(1, 1000), + IntervalDayTime::new(1, 2000), + IntervalDayTime::new(1, 3000), + IntervalDayTime::new(1, 4000), ]))); assert_vector_ref_eq(Arc::new(IntervalMonthDayNanoVector::from_values([ - 1000, 2000, 3000, 4000, + IntervalMonthDayNano::new(1, 1, 1000), + IntervalMonthDayNano::new(1, 1, 2000), + IntervalMonthDayNano::new(1, 1, 3000), + IntervalMonthDayNano::new(1, 1, 4000), ]))); assert_vector_ref_eq(Arc::new(DurationSecondVector::from_values([300, 310]))); assert_vector_ref_eq(Arc::new(DurationMillisecondVector::from_values([300, 310]))); @@ -302,12 +310,24 @@ mod tests { ); assert_vector_ref_ne( - Arc::new(IntervalDayTimeVector::from_values([1000, 2000])), - Arc::new(IntervalDayTimeVector::from_values([2100, 1200])), + Arc::new(IntervalDayTimeVector::from_values([ + IntervalDayTime::new(1, 1000), + IntervalDayTime::new(1, 2000), + ])), + Arc::new(IntervalDayTimeVector::from_values([ + IntervalDayTime::new(1, 2100), + IntervalDayTime::new(1, 1200), + ])), ); assert_vector_ref_ne( - Arc::new(IntervalMonthDayNanoVector::from_values([1000, 2000])), - Arc::new(IntervalMonthDayNanoVector::from_values([2100, 1200])), + Arc::new(IntervalMonthDayNanoVector::from_values([ + IntervalMonthDayNano::new(1, 1, 1000), + IntervalMonthDayNano::new(1, 1, 2000), + ])), + Arc::new(IntervalMonthDayNanoVector::from_values([ + IntervalMonthDayNano::new(1, 1, 2100), + IntervalMonthDayNano::new(1, 1, 1200), + ])), ); assert_vector_ref_ne( Arc::new(IntervalYearMonthVector::from_values([1000, 2000])), diff --git a/src/datatypes/src/vectors/helper.rs b/src/datatypes/src/vectors/helper.rs index ab176d126f..562dc55982 100644 --- a/src/datatypes/src/vectors/helper.rs +++ b/src/datatypes/src/vectors/helper.rs @@ -241,7 +241,10 @@ impl Helper { | ScalarValue::LargeList(_) | ScalarValue::Dictionary(_, _) | ScalarValue::Union(_, _, _) - | ScalarValue::Float16(_) => { + | ScalarValue::Float16(_) + | ScalarValue::Utf8View(_) + | ScalarValue::BinaryView(_) + | ScalarValue::Map(_) => { return error::ConversionSnafu { from: format!("Unsupported scalar value: {value}"), } @@ -415,13 +418,13 @@ mod tests { TimestampSecondArray, UInt16Array, UInt32Array, UInt64Array, UInt8Array, }; use arrow::buffer::Buffer; - use arrow::datatypes::Int32Type; + use arrow::datatypes::{Int32Type, IntervalMonthDayNano}; use arrow_array::{BinaryArray, DictionaryArray, FixedSizeBinaryArray, LargeStringArray}; use arrow_schema::DataType; use common_decimal::Decimal128; use common_time::time::Time; use common_time::timestamp::TimeUnit; - use common_time::{Date, DateTime, Duration, IntervalMonthDayNano}; + use common_time::{Date, DateTime, Duration}; use super::*; use crate::value::Value; @@ -509,6 +512,7 @@ mod tests { let value = ScalarValue::List(ScalarValue::new_list( &[ScalarValue::Int32(Some(1)), ScalarValue::Int32(Some(2))], &ArrowDataType::Int32, + true, )); let vector = Helper::try_from_scalar_value(value, 3).unwrap(); assert_eq!( @@ -679,9 +683,11 @@ mod tests { #[test] fn test_try_from_scalar_interval_value() { - let vector = - Helper::try_from_scalar_value(ScalarValue::IntervalMonthDayNano(Some(2000)), 3) - .unwrap(); + let vector = Helper::try_from_scalar_value( + ScalarValue::IntervalMonthDayNano(Some(IntervalMonthDayNano::new(1, 1, 2000))), + 3, + ) + .unwrap(); assert_eq!( ConcreteDataType::interval_month_day_nano_datatype(), @@ -690,7 +696,7 @@ mod tests { assert_eq!(3, vector.len()); for i in 0..vector.len() { assert_eq!( - Value::IntervalMonthDayNano(IntervalMonthDayNano::from_i128(2000)), + Value::IntervalMonthDayNano(IntervalMonthDayNano::new(1, 1, 2000).into()), vector.get(i) ); } diff --git a/src/datatypes/src/vectors/primitive.rs b/src/datatypes/src/vectors/primitive.rs index b61fedb1b9..6212276dd0 100644 --- a/src/datatypes/src/vectors/primitive.rs +++ b/src/datatypes/src/vectors/primitive.rs @@ -406,7 +406,7 @@ mod tests { Int32Array, Time32MillisecondArray, Time32SecondArray, Time64MicrosecondArray, Time64NanosecondArray, }; - use arrow::datatypes::DataType as ArrowDataType; + use arrow::datatypes::{DataType as ArrowDataType, IntervalDayTime}; use arrow_array::{ DurationMicrosecondArray, DurationMillisecondArray, DurationNanosecondArray, DurationSecondArray, IntervalDayTimeArray, IntervalYearMonthArray, @@ -650,10 +650,18 @@ mod tests { vector ); - let array: ArrayRef = Arc::new(IntervalDayTimeArray::from(vec![1000, 2000, 3000])); + let array: ArrayRef = Arc::new(IntervalDayTimeArray::from(vec![ + IntervalDayTime::new(1, 1000), + IntervalDayTime::new(1, 2000), + IntervalDayTime::new(1, 3000), + ])); let vector = IntervalDayTimeVector::try_from_arrow_array(array).unwrap(); assert_eq!( - IntervalDayTimeVector::from_values(vec![1000, 2000, 3000]), + IntervalDayTimeVector::from_values(vec![ + IntervalDayTime::new(1, 1000), + IntervalDayTime::new(1, 2000), + IntervalDayTime::new(1, 3000), + ]), vector ); diff --git a/src/flow/Cargo.toml b/src/flow/Cargo.toml index 6038d0f47a..b4545e1a89 100644 --- a/src/flow/Cargo.toml +++ b/src/flow/Cargo.toml @@ -37,6 +37,7 @@ datafusion.workspace = true datafusion-common.workspace = true datafusion-expr.workspace = true datafusion-physical-expr.workspace = true +datafusion-substrait.workspace = true datatypes.workspace = true enum-as-inner = "0.6.0" enum_dispatch = "0.3" diff --git a/src/flow/src/adapter/util.rs b/src/flow/src/adapter/util.rs index 6cedb7ac34..ffc447bace 100644 --- a/src/flow/src/adapter/util.rs +++ b/src/flow/src/adapter/util.rs @@ -185,12 +185,10 @@ pub fn table_info_value_to_relation_desc( pub fn from_proto_to_data_type( column_schema: &api::v1::ColumnSchema, ) -> Result { - let wrapper = ColumnDataTypeWrapper::try_new( - column_schema.datatype, - column_schema.datatype_extension.clone(), - ) - .map_err(BoxedError::new) - .context(ExternalSnafu)?; + let wrapper = + ColumnDataTypeWrapper::try_new(column_schema.datatype, column_schema.datatype_extension) + .map_err(BoxedError::new) + .context(ExternalSnafu)?; let cdt = ConcreteDataType::from(wrapper); Ok(cdt) diff --git a/src/flow/src/df_optimizer.rs b/src/flow/src/df_optimizer.rs index 76eb16ef38..59c45c0ade 100644 --- a/src/flow/src/df_optimizer.rs +++ b/src/flow/src/df_optimizer.rs @@ -23,6 +23,8 @@ use common_error::ext::BoxedError; use common_telemetry::debug; use datafusion::config::ConfigOptions; use datafusion::error::DataFusionError; +use datafusion::functions_aggregate::count::count_udaf; +use datafusion::functions_aggregate::sum::sum_udaf; use datafusion::optimizer::analyzer::count_wildcard_rule::CountWildcardRule; use datafusion::optimizer::analyzer::type_coercion::TypeCoercion; use datafusion::optimizer::common_subexpr_eliminate::CommonSubexprEliminate; @@ -35,8 +37,6 @@ use datafusion_common::tree_node::{ Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter, TreeNodeVisitor, }; use datafusion_common::{Column, DFSchema, ScalarValue}; -use datafusion_expr::aggregate_function::AggregateFunction; -use datafusion_expr::expr::AggregateFunctionDefinition; use datafusion_expr::utils::merge_schema; use datafusion_expr::{ BinaryExpr, Expr, Operator, Projection, ScalarUDFImpl, Signature, TypeSignature, Volatility, @@ -126,6 +126,7 @@ pub async fn sql_to_flow_plan( Ok(flow_plan) } +#[derive(Debug)] struct AvgExpandRule {} impl AvgExpandRule { @@ -238,7 +239,7 @@ fn put_aggr_to_proj_analyzer( fn expand_avg_analyzer( plan: datafusion_expr::LogicalPlan, ) -> Result, DataFusionError> { - let mut schema = merge_schema(plan.inputs()); + let mut schema = merge_schema(&plan.inputs()); if let datafusion_expr::LogicalPlan::TableScan(ts) = &plan { let source_schema = @@ -251,9 +252,10 @@ fn expand_avg_analyzer( let name_preserver = NamePreserver::new(&plan); // apply coercion rewrite all expressions in the plan individually plan.map_expressions(|expr| { - let original_name = name_preserver.save(&expr)?; - expr.rewrite(&mut expr_rewrite)? - .map_data(|expr| original_name.restore(expr)) + let original_name = name_preserver.save(&expr); + Ok(expr + .rewrite(&mut expr_rewrite)? + .update_data(|expr| original_name.restore(expr))) })? .map_data(|plan| plan.recompute_schema()) } @@ -280,12 +282,10 @@ impl TreeNodeRewriter for ExpandAvgRewriter<'_> { fn f_up(&mut self, expr: Expr) -> Result, DataFusionError> { if let Expr::AggregateFunction(aggr_func) = &expr { - if let AggregateFunctionDefinition::BuiltIn(AggregateFunction::Avg) = - &aggr_func.func_def - { + if aggr_func.func.name() == "avg" { let sum_expr = { let mut tmp = aggr_func.clone(); - tmp.func_def = AggregateFunctionDefinition::BuiltIn(AggregateFunction::Sum); + tmp.func = sum_udaf(); Expr::AggregateFunction(tmp) }; let sum_cast = { @@ -299,7 +299,7 @@ impl TreeNodeRewriter for ExpandAvgRewriter<'_> { let count_expr = { let mut tmp = aggr_func.clone(); - tmp.func_def = AggregateFunctionDefinition::BuiltIn(AggregateFunction::Count); + tmp.func = count_udaf(); Expr::AggregateFunction(tmp) }; @@ -329,6 +329,7 @@ impl TreeNodeRewriter for ExpandAvgRewriter<'_> { } /// expand tumble in aggr expr to tumble_start and tumble_end with column name like `window_start` +#[derive(Debug)] struct TumbleExpandRule {} impl TumbleExpandRule { @@ -528,6 +529,7 @@ impl ScalarUDFImpl for TumbleExpand { } /// This rule check all group by exprs, and make sure they are also in select clause in a aggr query +#[derive(Debug)] struct CheckGroupByRule {} impl CheckGroupByRule { diff --git a/src/flow/src/expr/func.rs b/src/flow/src/expr/func.rs index 4f0a944b16..8e23e74f32 100644 --- a/src/flow/src/expr/func.rs +++ b/src/flow/src/expr/func.rs @@ -829,18 +829,8 @@ impl BinaryFunc { arg_types: &[Option], ) -> Result<(Self, Signature), Error> { // this `name_to_op` if error simply return a similar message of `unsupported function xxx` so - let op = name_to_op(name).or_else(|err| { - if let datafusion_common::DataFusionError::NotImplemented(msg) = err { - InvalidQuerySnafu { - reason: format!("Unsupported binary function: {}", msg), - } - .fail() - } else { - InvalidQuerySnafu { - reason: format!("Error when parsing binary function: {:?}", err), - } - .fail() - } + let op = name_to_op(name).with_context(|| InvalidQuerySnafu { + reason: format!("Unsupported binary function: {}", name), })?; // get first arg type and make sure if both is some, they are the same diff --git a/src/flow/src/expr/relation/func.rs b/src/flow/src/expr/relation/func.rs index 29a0169229..0730e7d919 100644 --- a/src/flow/src/expr/relation/func.rs +++ b/src/flow/src/expr/relation/func.rs @@ -13,7 +13,6 @@ // limitations under the License. use std::collections::HashMap; -use std::str::FromStr; use std::sync::OnceLock; use datatypes::prelude::ConcreteDataType; @@ -21,10 +20,10 @@ use datatypes::value::Value; use datatypes::vectors::VectorRef; use serde::{Deserialize, Serialize}; use smallvec::smallvec; -use snafu::{IntoError, OptionExt}; +use snafu::OptionExt; use strum::{EnumIter, IntoEnumIterator}; -use crate::error::{DatafusionSnafu, Error, InvalidQuerySnafu}; +use crate::error::{Error, InvalidQuerySnafu}; use crate::expr::error::EvalError; use crate::expr::relation::accum::{Accum, Accumulator}; use crate::expr::signature::{GenericFn, Signature}; @@ -215,28 +214,14 @@ impl AggregateFunc { } spec }); - use datafusion_expr::aggregate_function::AggregateFunction as DfAggrFunc; - let df_aggr_func = DfAggrFunc::from_str(name).or_else(|err| { - if let datafusion_common::DataFusionError::NotImplemented(msg) = err { - InvalidQuerySnafu { - reason: format!("Unsupported aggregate function: {}", msg), - } - .fail() - } else { - Err(DatafusionSnafu { - context: "Error when parsing aggregate function", - } - .into_error(err)) - } - })?; - let generic_fn = match df_aggr_func { - DfAggrFunc::Max => GenericFn::Max, - DfAggrFunc::Min => GenericFn::Min, - DfAggrFunc::Sum => GenericFn::Sum, - DfAggrFunc::Count => GenericFn::Count, - DfAggrFunc::BoolOr => GenericFn::Any, - DfAggrFunc::BoolAnd => GenericFn::All, + let generic_fn = match name { + "max" => GenericFn::Max, + "min" => GenericFn::Min, + "sum" => GenericFn::Sum, + "count" => GenericFn::Count, + "bool_or" => GenericFn::Any, + "bool_and" => GenericFn::All, _ => { return InvalidQuerySnafu { reason: format!("Unknown aggregate function: {}", name), diff --git a/src/flow/src/plan.rs b/src/flow/src/plan.rs index 099954dc12..b2c91015e0 100644 --- a/src/flow/src/plan.rs +++ b/src/flow/src/plan.rs @@ -63,6 +63,7 @@ impl TypedPlan { pub fn projection(self, exprs: Vec) -> Result { let input_arity = self.schema.typ.column_types.len(); let output_arity = exprs.len(); + let (exprs, _expr_typs): (Vec<_>, Vec<_>) = exprs .into_iter() .map(|TypedExpr { expr, typ }| (expr, typ)) @@ -72,6 +73,7 @@ impl TypedPlan { .project(input_arity..input_arity + output_arity)? .into_safe(); let out_typ = self.schema.apply_mfp(&mfp)?; + let mfp = mfp.mfp; // special case for mfp to compose when the plan is already mfp let plan = match self.plan { diff --git a/src/flow/src/transform.rs b/src/flow/src/transform.rs index d6f30d3c88..01ff9cc299 100644 --- a/src/flow/src/transform.rs +++ b/src/flow/src/transform.rs @@ -13,10 +13,11 @@ // limitations under the License. //! Transform Substrait into execution plan -use std::collections::{BTreeMap, HashMap}; +use std::collections::BTreeMap; use std::sync::Arc; use common_error::ext::BoxedError; +use datafusion_substrait::extensions::Extensions; use datatypes::data_type::ConcreteDataType as CDT; use query::QueryEngine; use serde::{Deserialize, Serialize}; @@ -92,8 +93,15 @@ impl FunctionExtensions { self.anchor_to_name.get(anchor) } - pub fn inner_ref(&self) -> HashMap { - self.anchor_to_name.iter().map(|(k, v)| (*k, v)).collect() + pub fn to_extensions(&self) -> Extensions { + Extensions { + functions: self + .anchor_to_name + .iter() + .map(|(k, v)| (*k, v.clone())) + .collect(), + ..Default::default() + } } } @@ -179,6 +187,7 @@ mod test { pub fn create_test_ctx() -> FlownodeContext { let mut tri_map = IdToNameMap::new(); + // FIXME(discord9): deprecated, use `numbers_with_ts` instead since this table has no timestamp column { let gid = GlobalId::User(0); let name = [ diff --git a/src/flow/src/transform/aggr.rs b/src/flow/src/transform/aggr.rs index 27e6798e8d..27c7acfb1d 100644 --- a/src/flow/src/transform/aggr.rs +++ b/src/flow/src/transform/aggr.rs @@ -14,10 +14,10 @@ use itertools::Itertools; use snafu::OptionExt; +use substrait_proto::proto; use substrait_proto::proto::aggregate_function::AggregationInvocation; use substrait_proto::proto::aggregate_rel::{Grouping, Measure}; use substrait_proto::proto::function_argument::ArgType; -use substrait_proto::proto::{self}; use crate::error::{Error, NotImplementedSnafu, PlanSnafu}; use crate::expr::{ @@ -28,8 +28,11 @@ use crate::repr::{ColumnType, RelationDesc, RelationType}; use crate::transform::{substrait_proto, FlownodeContext, FunctionExtensions}; impl TypedExpr { + /// Allow `deprecated` due to the usage of deprecated grouping_expressions on datafusion to substrait side + #[allow(deprecated)] async fn from_substrait_agg_grouping( ctx: &mut FlownodeContext, + grouping_expressions: &[proto::Expression], groupings: &[Grouping], typ: &RelationDesc, extensions: &FunctionExtensions, @@ -38,7 +41,34 @@ impl TypedExpr { let mut group_expr = vec![]; match groupings.len() { 1 => { - for e in &groupings[0].grouping_expressions { + // handle case when deprecated grouping_expressions is referenced by index is empty + let expressions: Box + Send> = if groupings + [0] + .expression_references + .is_empty() + { + Box::new(groupings[0].grouping_expressions.iter()) + } else { + if groupings[0] + .expression_references + .iter() + .any(|idx| *idx as usize >= grouping_expressions.len()) + { + return PlanSnafu { + reason: format!("Invalid grouping expression reference: {:?} for grouping expr: {:?}", + groupings[0].expression_references, + grouping_expressions + ), + }.fail()?; + } + Box::new( + groupings[0] + .expression_references + .iter() + .map(|idx| &grouping_expressions[*idx as usize]), + ) + }; + for e in expressions { let x = TypedExpr::from_substrait_rex(e, typ, extensions).await?; group_expr.push(x); } @@ -251,9 +281,14 @@ impl TypedPlan { return not_impl_err!("Aggregate without an input is not supported"); }; - let group_exprs = - TypedExpr::from_substrait_agg_grouping(ctx, &agg.groupings, &input.schema, extensions) - .await?; + let group_exprs = TypedExpr::from_substrait_agg_grouping( + ctx, + &agg.grouping_expressions, + &agg.groupings, + &input.schema, + extensions, + ) + .await?; let time_index = find_time_index_in_group_exprs(&group_exprs); @@ -339,7 +374,6 @@ impl TypedPlan { reduce_plan: ReducePlan::Accumulable(accum_plan), }; // FIX(discord9): deal with key first - return Ok(TypedPlan { schema: output_type, plan, @@ -349,7 +383,6 @@ impl TypedPlan { #[cfg(test)] mod test { - use std::collections::BTreeMap; use std::time::Duration; use bytes::BytesMut; @@ -390,7 +423,7 @@ mod test { .with_key(vec![2]) .with_time_index(Some(1)) .into_named(vec![ - Some("SUM(abs(numbers_with_ts.number))".to_string()), + Some("sum(abs(numbers_with_ts.number))".to_string()), Some("window_start".to_string()), Some("window_end".to_string()), ]), @@ -454,14 +487,15 @@ mod test { false, )]) .into_unnamed(), - extensions: FunctionExtensions { - anchor_to_name: BTreeMap::from([ + extensions: FunctionExtensions::from_iter( + [ (0, "tumble_start".to_string()), (1, "tumble_end".to_string()), (2, "abs".to_string()), (3, "sum".to_string()), - ]), - }, + ] + .into_iter(), + ), }, ) .await @@ -530,7 +564,7 @@ mod test { .with_key(vec![2]) .with_time_index(Some(1)) .into_named(vec![ - Some("abs(SUM(numbers_with_ts.number))".to_string()), + Some("abs(sum(numbers_with_ts.number))".to_string()), Some("window_start".to_string()), Some("window_end".to_string()), ]), @@ -615,14 +649,15 @@ mod test { true, )]) .into_unnamed(), - extensions: FunctionExtensions { - anchor_to_name: BTreeMap::from([ + extensions: FunctionExtensions::from_iter( + [ (0, "abs".to_string()), (1, "tumble_start".to_string()), (2, "tumble_end".to_string()), (3, "sum".to_string()), - ]), - }, + ] + .into_iter(), + ), }) .await .unwrap(), @@ -784,8 +819,8 @@ mod test { .with_key(vec![0, 3]) .with_time_index(Some(2)) .into_named(vec![ - Some("numbers_with_ts.number".to_string()), - Some("AVG(numbers_with_ts.number)".to_string()), + Some("number".to_string()), + Some("avg(numbers_with_ts.number)".to_string()), Some("window_start".to_string()), Some("window_end".to_string()), ]), @@ -818,7 +853,7 @@ mod test { .with_key(vec![2]) .with_time_index(Some(1)) .into_named(vec![ - Some("SUM(numbers_with_ts.number)".to_string()), + Some("sum(numbers_with_ts.number)".to_string()), Some("window_start".to_string()), Some("window_end".to_string()), ]), @@ -928,7 +963,7 @@ mod test { .with_key(vec![2]) .with_time_index(Some(1)) .into_named(vec![ - Some("SUM(numbers_with_ts.number)".to_string()), + Some("sum(numbers_with_ts.number)".to_string()), Some("window_start".to_string()), Some("window_end".to_string()), ]), @@ -1060,8 +1095,8 @@ mod test { ]) .with_key(vec![1]) .into_named(vec![ - Some("AVG(numbers.number)".to_string()), - Some("numbers.number".to_string()), + Some("avg(numbers.number)".to_string()), + Some("number".to_string()), ]), plan: Plan::Mfp { input: Box::new( @@ -1192,7 +1227,7 @@ mod test { ); let expected = TypedPlan { schema: RelationType::new(vec![ColumnType::new(CDT::float64_datatype(), true)]) - .into_named(vec![Some("AVG(numbers.number)".to_string())]), + .into_named(vec![Some("avg(numbers.number)".to_string())]), plan: Plan::Mfp { input: Box::new( Plan::Reduce { @@ -1270,7 +1305,7 @@ mod test { }; let expected = TypedPlan { schema: RelationType::new(vec![ColumnType::new(CDT::uint64_datatype(), true)]) - .into_named(vec![Some("SUM(numbers.number)".to_string())]), + .into_named(vec![Some("sum(numbers.number)".to_string())]), plan: Plan::Reduce { input: Box::new( Plan::Get { @@ -1309,6 +1344,61 @@ mod test { assert_eq!(flow_plan.unwrap(), expected); } + #[tokio::test] + async fn test_distinct_number() { + let engine = create_test_query_engine(); + let sql = "SELECT DISTINCT number FROM numbers"; + let plan = sql_to_substrait(engine.clone(), sql).await; + + let mut ctx = create_test_ctx(); + let flow_plan = TypedPlan::from_substrait_plan(&mut ctx, &plan) + .await + .unwrap(); + + let expected = TypedPlan { + schema: RelationType::new(vec![ + ColumnType::new(CDT::uint32_datatype(), false), // col number + ]) + .with_key(vec![0]) + .into_named(vec![Some("number".to_string())]), + plan: Plan::Reduce { + input: Box::new( + Plan::Get { + id: crate::expr::Id::Global(GlobalId::User(0)), + } + .with_types( + RelationType::new(vec![ColumnType::new( + ConcreteDataType::uint32_datatype(), + false, + )]) + .into_named(vec![Some("number".to_string())]), + ) + .mfp(MapFilterProject::new(1).into_safe()) + .unwrap(), + ), + key_val_plan: KeyValPlan { + key_plan: MapFilterProject::new(1) + .map(vec![ScalarExpr::Column(0)]) + .unwrap() + .project(vec![1]) + .unwrap() + .into_safe(), + val_plan: MapFilterProject::new(1) + .project(vec![0]) + .unwrap() + .into_safe(), + }, + reduce_plan: ReducePlan::Accumulable(AccumulablePlan { + full_aggrs: vec![], + simple_aggrs: vec![], + distinct_aggrs: vec![], + }), + }, + }; + + assert_eq!(flow_plan, expected); + } + #[tokio::test] async fn test_sum_group_by() { let engine = create_test_query_engine(); @@ -1332,8 +1422,8 @@ mod test { ]) .with_key(vec![1]) .into_named(vec![ - Some("SUM(numbers.number)".to_string()), - Some("numbers.number".to_string()), + Some("sum(numbers.number)".to_string()), + Some("number".to_string()), ]), plan: Plan::Mfp { input: Box::new( @@ -1410,7 +1500,7 @@ mod test { let expected = TypedPlan { schema: RelationType::new(vec![ColumnType::new(CDT::uint64_datatype(), true)]) .into_named(vec![Some( - "SUM(numbers.number + numbers.number)".to_string(), + "sum(numbers.number + numbers.number)".to_string(), )]), plan: Plan::Reduce { input: Box::new( @@ -1490,7 +1580,7 @@ mod test { .with_time_index(Some(1)) .into_named(vec![ Some( - "MAX(numbers_with_ts.number) - MIN(numbers_with_ts.number) / Float64(30)" + "max(numbers_with_ts.number) - min(numbers_with_ts.number) / Float64(30)" .to_string(), ), Some("time_window".to_string()), @@ -1522,7 +1612,7 @@ mod test { df_scalar_fn: DfScalarFunction::try_from_raw_fn( RawDfScalarFn { f: BytesMut::from( - b"\x08\x02\"I\x1aG\nE\x8a\x02?\x08\x03\x12+\n\x17interval-month-day-nano\x12\x10\0\xac#\xfc\x06\0\0\0\0\0\0\0\0\0\0\0\x1a\x06\x12\x04:\x02\x10\x02\x1a\x06\x12\x04:\x02\x10\x02\x98\x03\x03\"\n\x1a\x08\x12\x06\n\x04\x12\x02\x08\x01".as_ref(), + b"\x08\x02\"\x0f\x1a\r\n\x0b\xa2\x02\x08\n\0\x12\x04\x10\x1e \t\"\n\x1a\x08\x12\x06\n\x04\x12\x02\x08\x01".as_ref(), ), input_schema: RelationType::new(vec![ColumnType::new( ConcreteDataType::interval_month_day_nano_datatype(), @@ -1532,15 +1622,13 @@ mod test { false, )]) .into_unnamed(), - extensions: FunctionExtensions { - anchor_to_name: BTreeMap::from([ + extensions: FunctionExtensions::from_iter([ (0, "subtract".to_string()), (1, "divide".to_string()), (2, "date_bin".to_string()), (3, "max".to_string()), (4, "min".to_string()), ]), - }, }, ) .await diff --git a/src/flow/src/transform/expr.rs b/src/flow/src/transform/expr.rs index de05b018ac..ed75252ee2 100644 --- a/src/flow/src/transform/expr.rs +++ b/src/flow/src/transform/expr.rs @@ -18,6 +18,8 @@ use std::sync::Arc; use common_error::ext::BoxedError; use common_telemetry::debug; +use datafusion::execution::SessionStateBuilder; +use datafusion::functions::all_default_functions; use datafusion_physical_expr::PhysicalExpr; use datatypes::data_type::ConcreteDataType as CDT; use snafu::{ensure, OptionExt, ResultExt}; @@ -86,16 +88,15 @@ pub(crate) async fn from_scalar_fn_to_df_fn_impl( }; let schema = input_schema.to_df_schema()?; - let df_expr = - // TODO(discord9): consider coloring everything async.... - substrait::df_logical_plan::consumer::from_substrait_rex( - &datafusion::prelude::SessionContext::new(), - &e, - &schema, - &extensions.inner_ref(), - ) - .await - ; + let df_expr = substrait::df_logical_plan::consumer::from_substrait_rex( + &SessionStateBuilder::new() + .with_scalar_functions(all_default_functions()) + .build(), + &e, + &schema, + &extensions.to_extensions(), + ) + .await; let expr = df_expr.context({ DatafusionSnafu { context: "Failed to convert substrait scalar function to datafusion scalar function", @@ -551,7 +552,8 @@ mod test { #[tokio::test] async fn test_where_and() { let engine = create_test_query_engine(); - let sql = "SELECT number FROM numbers WHERE number >= 1 AND number <= 3 AND number!=2"; + let sql = + "SELECT number FROM numbers_with_ts WHERE number >= 1 AND number <= 3 AND number!=2"; let plan = sql_to_substrait(engine.clone(), sql).await; let mut ctx = create_test_ctx(); @@ -561,37 +563,53 @@ mod test { let filter = ScalarExpr::CallVariadic { func: VariadicFunc::And, exprs: vec![ - ScalarExpr::Column(0).call_binary( - ScalarExpr::Literal(Value::from(1u32), CDT::uint32_datatype()), + ScalarExpr::Column(2).call_binary( + ScalarExpr::Literal(Value::from(1i64), CDT::int64_datatype()), BinaryFunc::Gte, ), - ScalarExpr::Column(0).call_binary( - ScalarExpr::Literal(Value::from(3u32), CDT::uint32_datatype()), + ScalarExpr::Column(2).call_binary( + ScalarExpr::Literal(Value::from(3i64), CDT::int64_datatype()), BinaryFunc::Lte, ), - ScalarExpr::Column(0).call_binary( - ScalarExpr::Literal(Value::from(2u32), CDT::uint32_datatype()), + ScalarExpr::Column(2).call_binary( + ScalarExpr::Literal(Value::from(2i64), CDT::int64_datatype()), BinaryFunc::NotEq, ), ], }; let expected = TypedPlan { schema: RelationType::new(vec![ColumnType::new(CDT::uint32_datatype(), false)]) - .into_named(vec![Some("numbers.number".to_string())]), + .into_named(vec![Some("number".to_string())]), plan: Plan::Mfp { input: Box::new( Plan::Get { - id: crate::expr::Id::Global(GlobalId::User(0)), + id: crate::expr::Id::Global(GlobalId::User(1)), } .with_types( - RelationType::new(vec![ColumnType::new( - ConcreteDataType::uint32_datatype(), - false, - )]) - .into_named(vec![Some("number".to_string())]), + RelationType::new(vec![ + ColumnType::new(ConcreteDataType::uint32_datatype(), false), + ColumnType::new( + ConcreteDataType::timestamp_millisecond_datatype(), + false, + ), + ]) + .into_named(vec![Some("number".to_string()), Some("ts".to_string())]), ), ), - mfp: MapFilterProject::new(1).filter(vec![filter]).unwrap(), + mfp: MapFilterProject::new(2) + .map(vec![ + ScalarExpr::CallUnary { + func: UnaryFunc::Cast(CDT::int64_datatype()), + expr: Box::new(ScalarExpr::Column(0)), + }, + ScalarExpr::Column(0), + ScalarExpr::Column(3), + ]) + .unwrap() + .filter(vec![filter]) + .unwrap() + .project(vec![4]) + .unwrap(), }, }; assert_eq!(flow_plan.unwrap(), expected); diff --git a/src/flow/src/transform/literal.rs b/src/flow/src/transform/literal.rs index ffb62ff14d..2640f606b2 100644 --- a/src/flow/src/transform/literal.rs +++ b/src/flow/src/transform/literal.rs @@ -16,29 +16,67 @@ use std::array::TryFromSliceError; use bytes::Bytes; use common_decimal::Decimal128; -use common_time::{Date, Timestamp}; +use common_time::timestamp::TimeUnit; +use common_time::{Date, IntervalMonthDayNano, Timestamp}; use datafusion_common::ScalarValue; use datatypes::data_type::ConcreteDataType as CDT; use datatypes::value::Value; use num_traits::FromBytes; -use snafu::ensure; -use substrait::substrait_proto_df::proto::expression::literal::user_defined::Val; -use substrait::substrait_proto_df::proto::expression::literal::UserDefined; +use snafu::OptionExt; use substrait::variation_const::{ DATE_32_TYPE_VARIATION_REF, DATE_64_TYPE_VARIATION_REF, DEFAULT_TYPE_VARIATION_REF, - INTERVAL_DAY_TIME_TYPE_REF, INTERVAL_DAY_TIME_TYPE_URL, INTERVAL_MONTH_DAY_NANO_TYPE_REF, - INTERVAL_MONTH_DAY_NANO_TYPE_URL, INTERVAL_YEAR_MONTH_TYPE_REF, INTERVAL_YEAR_MONTH_TYPE_URL, - TIMESTAMP_MICRO_TYPE_VARIATION_REF, TIMESTAMP_MILLI_TYPE_VARIATION_REF, - TIMESTAMP_NANO_TYPE_VARIATION_REF, TIMESTAMP_SECOND_TYPE_VARIATION_REF, UNSIGNED_INTEGER_TYPE_VARIATION_REF, }; -use substrait_proto::proto::expression::literal::LiteralType; +use substrait_proto::proto; +use substrait_proto::proto::expression::literal::{LiteralType, PrecisionTimestamp}; use substrait_proto::proto::expression::Literal; use substrait_proto::proto::r#type::Kind; use crate::error::{Error, NotImplementedSnafu, PlanSnafu, UnexpectedSnafu}; use crate::transform::substrait_proto; +#[derive(Debug)] +enum TimestampPrecision { + Second = 0, + Millisecond = 3, + Microsecond = 6, + Nanosecond = 9, +} + +impl TryFrom for TimestampPrecision { + type Error = Error; + + fn try_from(prec: i32) -> Result { + match prec { + 0 => Ok(Self::Second), + 3 => Ok(Self::Millisecond), + 6 => Ok(Self::Microsecond), + 9 => Ok(Self::Nanosecond), + _ => not_impl_err!("Unsupported precision: {prec}"), + } + } +} + +impl TimestampPrecision { + fn to_time_unit(&self) -> TimeUnit { + match self { + Self::Second => TimeUnit::Second, + Self::Millisecond => TimeUnit::Millisecond, + Self::Microsecond => TimeUnit::Microsecond, + Self::Nanosecond => TimeUnit::Nanosecond, + } + } + + fn to_cdt(&self) -> CDT { + match self { + Self::Second => CDT::timestamp_second_datatype(), + Self::Millisecond => CDT::timestamp_millisecond_datatype(), + Self::Microsecond => CDT::timestamp_microsecond_datatype(), + Self::Nanosecond => CDT::timestamp_nanosecond_datatype(), + } + } +} + /// TODO(discord9): this is copy from datafusion-substrait since the original function is not public, will be replace once is exported pub(crate) fn to_substrait_literal(value: &ScalarValue) -> Result { if value.is_null() { @@ -68,21 +106,34 @@ pub(crate) fn to_substrait_literal(value: &ScalarValue) -> Result (LiteralType::Fp32(*f), DEFAULT_TYPE_VARIATION_REF), ScalarValue::Float64(Some(f)) => (LiteralType::Fp64(*f), DEFAULT_TYPE_VARIATION_REF), + // TODO(discord9): deal with timezone ScalarValue::TimestampSecond(Some(t), _) => ( - LiteralType::Timestamp(*t), - TIMESTAMP_SECOND_TYPE_VARIATION_REF, + LiteralType::PrecisionTimestamp(PrecisionTimestamp { + value: *t, + precision: TimestampPrecision::Second as i32, + }), + DEFAULT_TYPE_VARIATION_REF, ), ScalarValue::TimestampMillisecond(Some(t), _) => ( - LiteralType::Timestamp(*t), - TIMESTAMP_MILLI_TYPE_VARIATION_REF, + LiteralType::PrecisionTimestamp(PrecisionTimestamp { + value: *t, + precision: TimestampPrecision::Millisecond as i32, + }), + DEFAULT_TYPE_VARIATION_REF, ), ScalarValue::TimestampMicrosecond(Some(t), _) => ( - LiteralType::Timestamp(*t), - TIMESTAMP_MICRO_TYPE_VARIATION_REF, + LiteralType::PrecisionTimestamp(PrecisionTimestamp { + value: *t, + precision: TimestampPrecision::Microsecond as i32, + }), + DEFAULT_TYPE_VARIATION_REF, ), ScalarValue::TimestampNanosecond(Some(t), _) => ( - LiteralType::Timestamp(*t), - TIMESTAMP_NANO_TYPE_VARIATION_REF, + LiteralType::PrecisionTimestamp(PrecisionTimestamp { + value: *t, + precision: TimestampPrecision::Nanosecond as i32, + }), + DEFAULT_TYPE_VARIATION_REF, ), ScalarValue::Date32(Some(d)) => (LiteralType::Date(*d), DATE_32_TYPE_VARIATION_REF), _ => ( @@ -124,25 +175,17 @@ pub(crate) fn from_substrait_literal(lit: &Literal) -> Result<(Value, CDT), Erro }, Some(LiteralType::Fp32(f)) => (Value::from(*f), CDT::float32_datatype()), Some(LiteralType::Fp64(f)) => (Value::from(*f), CDT::float64_datatype()), - Some(LiteralType::Timestamp(t)) => match lit.type_variation_reference { - TIMESTAMP_SECOND_TYPE_VARIATION_REF => ( - Value::from(Timestamp::new_second(*t)), - CDT::timestamp_second_datatype(), - ), - TIMESTAMP_MILLI_TYPE_VARIATION_REF => ( - Value::from(Timestamp::new_millisecond(*t)), - CDT::timestamp_millisecond_datatype(), - ), - TIMESTAMP_MICRO_TYPE_VARIATION_REF => ( - Value::from(Timestamp::new_microsecond(*t)), - CDT::timestamp_microsecond_datatype(), - ), - TIMESTAMP_NANO_TYPE_VARIATION_REF => ( - Value::from(Timestamp::new_nanosecond(*t)), - CDT::timestamp_nanosecond_datatype(), - ), - others => not_impl_err!("Unknown type variation reference {others}",)?, - }, + Some(LiteralType::Timestamp(t)) => ( + Value::from(Timestamp::new_microsecond(*t)), + CDT::timestamp_microsecond_datatype(), + ), + Some(LiteralType::PrecisionTimestamp(prec_ts)) => { + let (prec, val) = (prec_ts.precision, prec_ts.value); + let prec = TimestampPrecision::try_from(prec)?; + let unit = prec.to_time_unit(); + let typ = prec.to_cdt(); + (Value::from(Timestamp::new(val, unit)), typ) + } Some(LiteralType::Date(d)) => (Value::from(Date::new(*d)), CDT::date_datatype()), Some(LiteralType::String(s)) => (Value::from(s.clone()), CDT::string_datatype()), Some(LiteralType::Binary(b)) | Some(LiteralType::FixedBinary(b)) => { @@ -174,30 +217,116 @@ pub(crate) fn from_substrait_literal(lit: &Literal) -> Result<(Value, CDT), Erro ) } Some(LiteralType::Null(ntype)) => (Value::Null, from_substrait_type(ntype)?), - Some(LiteralType::IntervalDayToSecond(interval)) => { - let (days, seconds, microseconds) = - (interval.days, interval.seconds, interval.microseconds); - let millis = microseconds / 1000 + seconds * 1000; - let value_interval = common_time::IntervalDayTime::new(days, millis); + Some(LiteralType::IntervalDayToSecond(interval)) => from_interval_day_sec(interval)?, + Some(LiteralType::IntervalYearToMonth(interval)) => from_interval_year_month(interval)?, + Some(LiteralType::IntervalCompound(interval_compound)) => { + let interval_day_time = &interval_compound + .interval_day_to_second + .map(|i| from_interval_day_sec(&i)) + .transpose()?; + let interval_year_month = &interval_compound + .interval_year_to_month + .map(|i| from_interval_year_month(&i)) + .transpose()?; + let mut compound = IntervalMonthDayNano::new(0, 0, 0); + if let Some(day_sec) = interval_day_time { + let Value::IntervalDayTime(day_time) = day_sec.0 else { + UnexpectedSnafu { + reason: format!("Expect IntervalDayTime, found {:?}", day_sec), + } + .fail()? + }; + // 1 day in milliseconds = 24 * 60 * 60 * 1000 = 8.64e7 ms = 8.64e13 ns << 2^63 + // so overflow is unexpected + compound.nanoseconds = compound + .nanoseconds + .checked_add(day_time.milliseconds as i64 * 1_000_000) + .with_context(|| UnexpectedSnafu { + reason: format!( + "Overflow when converting interval: {:?}", + interval_compound + ), + })?; + compound.days += day_time.days; + } + + if let Some(year_month) = interval_year_month { + let Value::IntervalYearMonth(year_month) = year_month.0 else { + UnexpectedSnafu { + reason: format!("Expect IntervalYearMonth, found {:?}", year_month), + } + .fail()? + }; + compound.months += year_month.months; + } + ( - Value::IntervalDayTime(value_interval), - CDT::interval_day_time_datatype(), + Value::IntervalMonthDayNano(compound), + CDT::interval_month_day_nano_datatype(), ) } - Some(LiteralType::IntervalYearToMonth(interval)) => ( - Value::IntervalYearMonth(common_time::IntervalYearMonth::new( - interval.years * 12 + interval.months, - )), - CDT::interval_year_month_datatype(), - ), - Some(LiteralType::UserDefined(user_defined)) => { - from_substrait_user_defined_type(user_defined)? - } _ => not_impl_err!("unsupported literal_type: {:?}", &lit.literal_type)?, }; Ok(scalar_value) } +fn from_interval_day_sec( + interval: &proto::expression::literal::IntervalDayToSecond, +) -> Result<(Value, CDT), Error> { + let (days, seconds, subseconds) = (interval.days, interval.seconds, interval.subseconds); + let millis = if let Some(prec) = interval.precision_mode { + use substrait_proto::proto::expression::literal::interval_day_to_second::PrecisionMode; + match prec { + PrecisionMode::Precision(e) => { + if e >= 3 { + subseconds + / 10_i64 + .checked_pow((e - 3) as _) + .with_context(|| UnexpectedSnafu { + reason: format!( + "Overflow when converting interval: {:?}", + interval + ), + })? + } else { + subseconds + * 10_i64 + .checked_pow((3 - e) as _) + .with_context(|| UnexpectedSnafu { + reason: format!( + "Overflow when converting interval: {:?}", + interval + ), + })? + } + } + PrecisionMode::Microseconds(_) => subseconds / 1000, + } + } else if subseconds == 0 { + 0 + } else { + not_impl_err!("unsupported subseconds without precision_mode: {subseconds}")? + }; + + let value_interval = common_time::IntervalDayTime::new(days, seconds * 1000 + millis as i32); + + Ok(( + Value::IntervalDayTime(value_interval), + CDT::interval_day_time_datatype(), + )) +} + +fn from_interval_year_month( + interval: &proto::expression::literal::IntervalYearToMonth, +) -> Result<(Value, CDT), Error> { + let value_interval = common_time::IntervalYearMonth::new(interval.years * 12 + interval.months); + + Ok(( + Value::IntervalYearMonth(value_interval), + CDT::interval_year_month_datatype(), + )) +} + fn from_bytes(i: &Bytes) -> Result where for<'a> &'a ::Bytes: @@ -218,79 +347,6 @@ where Ok(i) } -fn from_substrait_user_defined_type(user_defined: &UserDefined) -> Result<(Value, CDT), Error> { - if let UserDefined { - type_reference, - type_parameters: _, - val: Some(Val::Value(val)), - } = user_defined - { - // see https://github.com/apache/datafusion/blob/146b679aa19c7749cc73d0c27440419d6498142b/datafusion/substrait/src/logical_plan/producer.rs#L1957 - // for interval type's transform to substrait - let ret = match *type_reference { - INTERVAL_YEAR_MONTH_TYPE_REF => { - ensure!( - val.type_url == INTERVAL_YEAR_MONTH_TYPE_URL, - UnexpectedSnafu { - reason: format!( - "Expect {}, found {} in type_url", - INTERVAL_YEAR_MONTH_TYPE_URL, val.type_url - ) - } - ); - let i: i32 = from_bytes(&val.value)?; - let value_interval = common_time::IntervalYearMonth::new(i); - ( - Value::IntervalYearMonth(value_interval), - CDT::interval_year_month_datatype(), - ) - } - INTERVAL_MONTH_DAY_NANO_TYPE_REF => { - ensure!( - val.type_url == INTERVAL_MONTH_DAY_NANO_TYPE_URL, - UnexpectedSnafu { - reason: format!( - "Expect {}, found {} in type_url", - INTERVAL_MONTH_DAY_NANO_TYPE_URL, val.type_url - ) - } - ); - // TODO(yingwen): Datafusion may change the representation of the interval type. - let i: i128 = from_bytes(&val.value)?; - let (months, days, nsecs) = ((i >> 96) as i32, (i >> 64) as i32, i as i64); - let value_interval = common_time::IntervalMonthDayNano::new(months, days, nsecs); - ( - Value::IntervalMonthDayNano(value_interval), - CDT::interval_month_day_nano_datatype(), - ) - } - INTERVAL_DAY_TIME_TYPE_REF => { - ensure!( - val.type_url == INTERVAL_DAY_TIME_TYPE_URL, - UnexpectedSnafu { - reason: format!( - "Expect {}, found {} in type_url", - INTERVAL_DAY_TIME_TYPE_URL, val.type_url - ) - } - ); - // TODO(yingwen): Datafusion may change the representation of the interval type. - let i: i64 = from_bytes(&val.value)?; - let (days, millis) = ((i >> 32) as i32, i as i32); - let value_interval = common_time::IntervalDayTime::new(days, millis); - ( - Value::IntervalDayTime(value_interval), - CDT::interval_day_time_datatype(), - ) - } - _ => return not_impl_err!("unsupported user defined type: {:?}", user_defined)?, - }; - Ok(ret) - } else { - not_impl_err!("Expect val to be Some(...)") - } -} - /// convert a Substrait type into a ConcreteDataType pub fn from_substrait_type(null_type: &substrait_proto::proto::Type) -> Result { if let Some(kind) = &null_type.kind { @@ -318,13 +374,9 @@ pub fn from_substrait_type(null_type: &substrait_proto::proto::Type) -> Result Ok(CDT::float32_datatype()), Kind::Fp64(_) => Ok(CDT::float64_datatype()), - Kind::Timestamp(ts) => match ts.type_variation_reference { - TIMESTAMP_SECOND_TYPE_VARIATION_REF => Ok(CDT::timestamp_second_datatype()), - TIMESTAMP_MILLI_TYPE_VARIATION_REF => Ok(CDT::timestamp_millisecond_datatype()), - TIMESTAMP_MICRO_TYPE_VARIATION_REF => Ok(CDT::timestamp_microsecond_datatype()), - TIMESTAMP_NANO_TYPE_VARIATION_REF => Ok(CDT::timestamp_nanosecond_datatype()), - v => not_impl_err!("Unsupported Substrait type variation {v} of type {kind:?}"), - }, + Kind::PrecisionTimestamp(ts) => { + Ok(TimestampPrecision::try_from(ts.precision)?.to_cdt()) + } Kind::Date(date) => match date.type_variation_reference { DATE_32_TYPE_VARIATION_REF | DATE_64_TYPE_VARIATION_REF => Ok(CDT::date_datatype()), v => not_impl_err!("Unsupported Substrait type variation {v} of type {kind:?}"), diff --git a/src/flow/src/transform/plan.rs b/src/flow/src/transform/plan.rs index 5e8599f368..bccfd2850a 100644 --- a/src/flow/src/transform/plan.rs +++ b/src/flow/src/transform/plan.rs @@ -83,6 +83,7 @@ impl TypedPlan { // because this `input.schema` is incorrect for pre-expand substrait plan, so we have to use schema before expand multi-value // function to correctly transform it, and late rewrite it + // TODO(discord9): this logic is obsoleted since now expand happens in datafusion optimizer let schema_before_expand = { let input_schema = input.schema.clone(); let auto_columns: HashSet = @@ -176,6 +177,7 @@ impl TypedPlan { } .fail()?, }; + let table = ctx.table(&table_reference).await?; let get_table = Plan::Get { id: crate::expr::Id::Global(table.0), @@ -252,7 +254,7 @@ mod test { let expected = TypedPlan { schema: RelationType::new(vec![ColumnType::new(CDT::uint32_datatype(), false)]) - .into_named(vec![Some("numbers.number".to_string())]), + .into_named(vec![Some("number".to_string())]), plan: Plan::Mfp { input: Box::new( Plan::Get { diff --git a/src/index/src/bloom_filter/applier.rs b/src/index/src/bloom_filter/applier.rs index c60b99d008..e87a94cd1b 100644 --- a/src/index/src/bloom_filter/applier.rs +++ b/src/index/src/bloom_filter/applier.rs @@ -50,7 +50,7 @@ impl BloomFilterApplier { let deduped_locs = locs .iter() .dedup() - .map(|i| self.meta.bloom_filter_locs[*i as usize].clone()) + .map(|i| self.meta.bloom_filter_locs[*i as usize]) .collect::>(); let bfs = self.reader.bloom_filter_vec(&deduped_locs).await?; diff --git a/src/meta-srv/Cargo.toml b/src/meta-srv/Cargo.toml index d875095f75..1da3bbb2bf 100644 --- a/src/meta-srv/Cargo.toml +++ b/src/meta-srv/Cargo.toml @@ -17,6 +17,7 @@ local-ip-address.workspace = true [dependencies] api.workspace = true async-trait = "0.1" +bytes.workspace = true chrono.workspace = true clap.workspace = true client.workspace = true @@ -43,9 +44,10 @@ derive_builder.workspace = true etcd-client.workspace = true futures.workspace = true h2 = "0.3" -http-body = "0.4" +http-body-util = "0.1" humantime.workspace = true humantime-serde.workspace = true +hyper-util = { workspace = true, features = ["tokio"] } itertools.workspace = true lazy_static.workspace = true once_cell.workspace = true diff --git a/src/meta-srv/src/mocks.rs b/src/meta-srv/src/mocks.rs index 2a621814f2..656ceeb3d9 100644 --- a/src/meta-srv/src/mocks.rs +++ b/src/meta-srv/src/mocks.rs @@ -25,6 +25,7 @@ use common_meta::key::TableMetadataManager; use common_meta::kv_backend::etcd::EtcdStore; use common_meta::kv_backend::memory::MemoryKvBackend; use common_meta::kv_backend::{KvBackendRef, ResettableKvBackendRef}; +use hyper_util::rt::TokioIo; use tonic::codec::CompressionEncoding; use tower::service_fn; @@ -138,7 +139,7 @@ pub async fn mock( async move { if let Some(client) = client { - Ok(client) + Ok(TokioIo::new(client)) } else { Err(std::io::Error::new( std::io::ErrorKind::Other, diff --git a/src/meta-srv/src/procedure/utils.rs b/src/meta-srv/src/procedure/utils.rs index 51db562a32..d81c9d9c97 100644 --- a/src/meta-srv/src/procedure/utils.rs +++ b/src/meta-srv/src/procedure/utils.rs @@ -26,6 +26,7 @@ pub mod mock { use common_meta::peer::Peer; use common_runtime::runtime::BuilderBuild; use common_runtime::{Builder as RuntimeBuilder, Runtime}; + use hyper_util::rt::TokioIo; use servers::grpc::region_server::{RegionServerHandler, RegionServerRequestHandler}; use tokio::sync::mpsc; use tonic::codec::CompressionEncoding; @@ -77,7 +78,7 @@ pub mod mock { datanode.addr.clone(), service_fn(move |_| { let client = client.take().unwrap(); - async move { Ok::<_, Error>(client) } + async move { Ok::<_, Error>(TokioIo::new(client)) } }), ) .unwrap(); diff --git a/src/meta-srv/src/service/admin.rs b/src/meta-srv/src/service/admin.rs index 772c3bdba5..7187d87447 100644 --- a/src/meta-srv/src/service/admin.rs +++ b/src/meta-srv/src/service/admin.rs @@ -24,6 +24,8 @@ use std::convert::Infallible; use std::sync::Arc; use std::task::{Context, Poll}; +use bytes::Bytes; +use http_body_util::{BodyExt, Full}; use tonic::body::BoxBody; use tonic::codegen::{empty_body, http, BoxFuture, Service}; use tonic::server::NamedService; @@ -190,10 +192,12 @@ fn check_path(path: &str) { } } +/// Returns a [BoxBody] from a string. +/// The implementation follows [empty_body()]. fn boxed(body: String) -> BoxBody { - use http_body::Body; - - body.map_err(|_| panic!("")).boxed_unsync() + Full::new(Bytes::from(body)) + .map_err(|err| match err {}) + .boxed_unsync() } #[cfg(test)] diff --git a/src/mito2/src/cache/cache_size.rs b/src/mito2/src/cache/cache_size.rs index a60b203bd7..63e7b88cfa 100644 --- a/src/mito2/src/cache/cache_size.rs +++ b/src/mito2/src/cache/cache_size.rs @@ -120,7 +120,8 @@ fn parquet_offset_index_heap_size(offset_index: &ParquetOffsetIndex) -> usize { row_group .iter() .map(|column| { - column.len() * mem::size_of::() + mem::size_of_val(column) + column.page_locations.len() * mem::size_of::() + + mem::size_of_val(column) }) .sum::() + mem::size_of_val(row_group) diff --git a/src/mito2/src/cache/file_cache.rs b/src/mito2/src/cache/file_cache.rs index be05bfc98e..b64968ee54 100644 --- a/src/mito2/src/cache/file_cache.rs +++ b/src/mito2/src/cache/file_cache.rs @@ -411,7 +411,7 @@ mod tests { let cache = FileCache::new( local_store.clone(), ReadableSize::mb(10), - Some(Duration::from_millis(5)), + Some(Duration::from_millis(10)), ); let region_id = RegionId::new(2000, 0); let file_id = FileId::random(); @@ -437,7 +437,7 @@ mod tests { let exist = cache.reader(key).await; assert!(exist.is_some()); - tokio::time::sleep(Duration::from_millis(10)).await; + tokio::time::sleep(Duration::from_millis(15)).await; cache.memory_index.run_pending_tasks().await; let non = cache.reader(key).await; assert!(non.is_none()); diff --git a/src/mito2/src/engine/basic_test.rs b/src/mito2/src/engine/basic_test.rs index 43409a2c9c..3aa9223c6e 100644 --- a/src/mito2/src/engine/basic_test.rs +++ b/src/mito2/src/engine/basic_test.rs @@ -556,7 +556,7 @@ async fn test_region_usage() { // region is empty now, check manifest size let region = engine.get_region(region_id).unwrap(); let region_stat = region.region_statistic(); - assert_eq!(region_stat.manifest_size, 717); + assert!(region_stat.manifest_size > 0); // put some rows let rows = Rows { @@ -583,12 +583,12 @@ async fn test_region_usage() { flush_region(&engine, region_id, None).await; let region_stat = region.region_statistic(); - assert!(region_stat.sst_size > 0); // Chief says this assert can ensure the size is counted. + assert!(region_stat.sst_size > 0); assert_eq!(region_stat.num_rows, 10); // region total usage // Some memtables may share items. - assert!(region_stat.estimated_disk_size() >= 4028); + assert!(region_stat.estimated_disk_size() > 3000); } #[tokio::test] diff --git a/src/mito2/src/memtable/bulk/row_group_reader.rs b/src/mito2/src/memtable/bulk/row_group_reader.rs index 14c3fbe68b..3c1fd3e1a1 100644 --- a/src/mito2/src/memtable/bulk/row_group_reader.rs +++ b/src/mito2/src/memtable/bulk/row_group_reader.rs @@ -47,32 +47,20 @@ impl<'a> MemtableRowGroupPageFetcher<'a> { parquet_meta: &'a ParquetMetaData, bytes: Bytes, ) -> Self { - let metadata = parquet_meta.row_group(row_group_idx); - let row_count = metadata.num_rows() as usize; - let page_locations = parquet_meta - .offset_index() - .map(|x| x[row_group_idx].as_slice()); - Self { - base: RowGroupBase { - metadata, - page_locations, - row_count, - column_chunks: vec![None; metadata.columns().len()], - // the cached `column_uncompressed_pages` would never be used in Memtable readers. - column_uncompressed_pages: vec![None; metadata.columns().len()], - }, + // the cached `column_uncompressed_pages` would never be used in Memtable readers. + base: RowGroupBase::new(parquet_meta, row_group_idx), bytes, } } /// Fetches column pages from memory file. pub(crate) fn fetch(&mut self, projection: &ProjectionMask, selection: Option<&RowSelection>) { - if let Some((selection, page_locations)) = selection.zip(self.base.page_locations) { + if let Some((selection, offset_index)) = selection.zip(self.base.offset_index) { // Selection provided. let (fetch_ranges, page_start_offsets) = self.base - .calc_sparse_read_ranges(projection, page_locations, selection); + .calc_sparse_read_ranges(projection, offset_index, selection); if fetch_ranges.is_empty() { return; } diff --git a/src/mito2/src/read/projection.rs b/src/mito2/src/read/projection.rs index 3a4ade9c19..4ffc021e42 100644 --- a/src/mito2/src/read/projection.rs +++ b/src/mito2/src/read/projection.rs @@ -55,15 +55,27 @@ pub struct ProjectionMapper { column_ids: Vec, /// Ids and DataTypes of field columns in the [Batch]. batch_fields: Vec<(ColumnId, ConcreteDataType)>, + /// `true` If the original projection is empty. + is_empty_projection: bool, } impl ProjectionMapper { /// Returns a new mapper with projection. + /// If `projection` is empty, it outputs [RecordBatch] without any column but only a row count. + /// `SELECT COUNT(*) FROM table` is an example that uses an empty projection. DataFusion accepts + /// empty `RecordBatch` and only use its row count in this query. pub fn new( metadata: &RegionMetadataRef, projection: impl Iterator, ) -> Result { - let projection: Vec<_> = projection.collect(); + let mut projection: Vec<_> = projection.collect(); + // If the original projection is empty. + let is_empty_projection = projection.is_empty(); + if is_empty_projection { + // If the projection is empty, we still read the time index column. + projection.push(metadata.time_index_column_pos()); + } + let mut column_schemas = Vec::with_capacity(projection.len()); let mut column_ids = Vec::with_capacity(projection.len()); for idx in &projection { @@ -81,6 +93,21 @@ impl ProjectionMapper { column_schemas.push(metadata.schema.column_schemas()[*idx].clone()); } let codec = DensePrimaryKeyCodec::new(metadata); + + if is_empty_projection { + // If projection is empty, we don't output any column. + return Ok(ProjectionMapper { + metadata: metadata.clone(), + batch_indices: vec![], + has_tags: false, + codec, + output_schema: Arc::new(Schema::new(vec![])), + column_ids, + batch_fields: vec![], + is_empty_projection, + }); + } + // Safety: Columns come from existing schema. let output_schema = Arc::new(Schema::new(column_schemas)); // Get fields in each batch. @@ -127,6 +154,7 @@ impl ProjectionMapper { output_schema, column_ids, batch_fields, + is_empty_projection, }) } @@ -140,7 +168,8 @@ impl ProjectionMapper { &self.metadata } - /// Returns ids of projected columns. + /// Returns ids of projected columns that we need to read + /// from memtables and SSTs. pub(crate) fn column_ids(&self) -> &[ColumnId] { &self.column_ids } @@ -151,6 +180,8 @@ impl ProjectionMapper { } /// Returns the schema of converted [RecordBatch]. + /// This is the schema that the stream will output. This schema may contain + /// less columns than [ProjectionMapper::column_ids()]. pub(crate) fn output_schema(&self) -> SchemaRef { self.output_schema.clone() } @@ -168,6 +199,10 @@ impl ProjectionMapper { batch: &Batch, cache_strategy: &CacheStrategy, ) -> common_recordbatch::error::Result { + if self.is_empty_projection { + return RecordBatch::new_with_count(self.output_schema.clone(), batch.num_rows()); + } + debug_assert_eq!(self.batch_fields.len(), batch.fields().len()); debug_assert!(self .batch_fields @@ -411,4 +446,30 @@ mod tests { +----+----+"; assert_eq!(expect, print_record_batch(record_batch)); } + + #[test] + fn test_projection_mapper_empty_projection() { + let metadata = Arc::new( + TestRegionMetadataBuilder::default() + .num_tags(2) + .num_fields(2) + .build(), + ); + // Empty projection + let mapper = ProjectionMapper::new(&metadata, [].into_iter()).unwrap(); + assert_eq!([0], mapper.column_ids()); // Should still read the time index column + assert!(mapper.batch_fields().is_empty()); + assert!(!mapper.has_tags); + assert!(mapper.batch_indices.is_empty()); + assert!(mapper.output_schema().is_empty()); + assert!(mapper.is_empty_projection); + + let batch = new_batch(0, &[1, 2], &[], 3); + let cache = CacheManager::builder().vector_cache_size(1024).build(); + let cache = CacheStrategy::EnableAll(Arc::new(cache)); + let record_batch = mapper.convert(&batch, &cache).unwrap(); + assert_eq!(3, record_batch.num_rows()); + assert_eq!(0, record_batch.num_columns()); + assert!(record_batch.schema.is_empty()); + } } diff --git a/src/mito2/src/request.rs b/src/mito2/src/request.rs index 0cd59aa1d9..19add6b3db 100644 --- a/src/mito2/src/request.rs +++ b/src/mito2/src/request.rs @@ -155,7 +155,7 @@ impl WriteRequest { ensure!( is_column_type_value_eq( input_col.datatype, - input_col.datatype_extension.clone(), + input_col.datatype_extension, &column.column_schema.data_type ), InvalidRequestSnafu { diff --git a/src/mito2/src/sst/parquet/file_range.rs b/src/mito2/src/sst/parquet/file_range.rs index 607a15ac88..388dc24677 100644 --- a/src/mito2/src/sst/parquet/file_range.rs +++ b/src/mito2/src/sst/parquet/file_range.rs @@ -215,22 +215,18 @@ impl FileRangeContext { let stats = column_metadata.statistics().context(StatsNotPresentSnafu { file_path: self.reader_builder.file_path(), })?; - if stats.has_min_max_set() { - stats - .min_bytes() - .try_into() - .map(i32::from_le_bytes) - .map(|min_op_type| min_op_type == OpType::Delete as i32) - .ok() - .context(DecodeStatsSnafu { - file_path: self.reader_builder.file_path(), - }) - } else { - DecodeStatsSnafu { + stats + .min_bytes_opt() + .context(StatsNotPresentSnafu { file_path: self.reader_builder.file_path(), - } - .fail() - } + })? + .try_into() + .map(i32::from_le_bytes) + .map(|min_op_type| min_op_type == OpType::Delete as i32) + .ok() + .context(DecodeStatsSnafu { + file_path: self.reader_builder.file_path(), + }) } } diff --git a/src/mito2/src/sst/parquet/format.rs b/src/mito2/src/sst/parquet/format.rs index e10d29d7ac..34a1da565e 100644 --- a/src/mito2/src/sst/parquet/format.rs +++ b/src/mito2/src/sst/parquet/format.rs @@ -410,9 +410,6 @@ impl ReadFormat { .borrow() .column(self.primary_key_position()) .statistics()?; - if !stats.has_min_max_set() { - return None; - } match stats { Statistics::Boolean(_) => None, Statistics::Int32(_) => None, @@ -421,7 +418,11 @@ impl ReadFormat { Statistics::Float(_) => None, Statistics::Double(_) => None, Statistics::ByteArray(s) => { - let bytes = if is_min { s.min_bytes() } else { s.max_bytes() }; + let bytes = if is_min { + s.min_bytes_opt()? + } else { + s.max_bytes_opt()? + }; converter.decode_leftmost(bytes).ok()? } Statistics::FixedLenByteArray(_) => None, @@ -460,39 +461,40 @@ impl ReadFormat { .iter() .map(|meta| { let stats = meta.borrow().column(column_index).statistics()?; - if !stats.has_min_max_set() { - return None; - } match stats { Statistics::Boolean(s) => Some(ScalarValue::Boolean(Some(if is_min { - *s.min() + *s.min_opt()? } else { - *s.max() + *s.max_opt()? }))), Statistics::Int32(s) => Some(ScalarValue::Int32(Some(if is_min { - *s.min() + *s.min_opt()? } else { - *s.max() + *s.max_opt()? }))), Statistics::Int64(s) => Some(ScalarValue::Int64(Some(if is_min { - *s.min() + *s.min_opt()? } else { - *s.max() + *s.max_opt()? }))), Statistics::Int96(_) => None, Statistics::Float(s) => Some(ScalarValue::Float32(Some(if is_min { - *s.min() + *s.min_opt()? } else { - *s.max() + *s.max_opt()? }))), Statistics::Double(s) => Some(ScalarValue::Float64(Some(if is_min { - *s.min() + *s.min_opt()? } else { - *s.max() + *s.max_opt()? }))), Statistics::ByteArray(s) => { - let bytes = if is_min { s.min_bytes() } else { s.max_bytes() }; + let bytes = if is_min { + s.min_bytes_opt()? + } else { + s.max_bytes_opt()? + }; let s = String::from_utf8(bytes.to_vec()).ok(); Some(ScalarValue::Utf8(s)) } @@ -514,7 +516,7 @@ impl ReadFormat { let values = row_groups.iter().map(|meta| { let col = meta.borrow().column(column_index); let stat = col.statistics()?; - Some(stat.null_count()) + stat.null_count_opt() }); Some(Arc::new(UInt64Array::from_iter(values))) } @@ -595,31 +597,30 @@ pub(crate) fn parquet_row_group_time_range( let time_index_pos = num_columns - FIXED_POS_COLUMN_NUM; let stats = row_group_meta.column(time_index_pos).statistics()?; - if stats.has_min_max_set() { - // The physical type for the timestamp should be i64. - let (min, max) = match stats { - Statistics::Int64(value_stats) => (*value_stats.min(), *value_stats.max()), - Statistics::Int32(_) - | Statistics::Boolean(_) - | Statistics::Int96(_) - | Statistics::Float(_) - | Statistics::Double(_) - | Statistics::ByteArray(_) - | Statistics::FixedLenByteArray(_) => return None, - }; + // The physical type for the timestamp should be i64. + let (min, max) = match stats { + Statistics::Int64(value_stats) => (*value_stats.min_opt()?, *value_stats.max_opt()?), + Statistics::Int32(_) + | Statistics::Boolean(_) + | Statistics::Int96(_) + | Statistics::Float(_) + | Statistics::Double(_) + | Statistics::ByteArray(_) + | Statistics::FixedLenByteArray(_) => { + common_telemetry::warn!( + "Invalid statistics {:?} for time index in parquet in {}", + stats, + file_meta.file_id + ); + return None; + } + }; - debug_assert!( - min >= file_meta.time_range.0.value() && min <= file_meta.time_range.1.value() - ); - debug_assert!( - max >= file_meta.time_range.0.value() && max <= file_meta.time_range.1.value() - ); - let unit = file_meta.time_range.0.unit(); + debug_assert!(min >= file_meta.time_range.0.value() && min <= file_meta.time_range.1.value()); + debug_assert!(max >= file_meta.time_range.0.value() && max <= file_meta.time_range.1.value()); + let unit = file_meta.time_range.0.unit(); - Some((Timestamp::new(min, unit), Timestamp::new(max, unit))) - } else { - None - } + Some((Timestamp::new(min, unit), Timestamp::new(max, unit))) } #[cfg(test)] diff --git a/src/mito2/src/sst/parquet/metadata.rs b/src/mito2/src/sst/parquet/metadata.rs index e39f259723..26059c4d23 100644 --- a/src/mito2/src/sst/parquet/metadata.rs +++ b/src/mito2/src/sst/parquet/metadata.rs @@ -13,8 +13,7 @@ // limitations under the License. use object_store::ObjectStore; -use parquet::file::footer::{decode_footer, decode_metadata}; -use parquet::file::metadata::ParquetMetaData; +use parquet::file::metadata::{ParquetMetaData, ParquetMetaDataReader}; use parquet::file::FOOTER_SIZE; use snafu::ResultExt; @@ -96,7 +95,7 @@ impl<'a> MetadataLoader<'a> { let mut footer = [0; 8]; footer.copy_from_slice(&buffer[buffer_len - FOOTER_SIZE..]); - let metadata_len = decode_footer(&footer).map_err(|e| { + let metadata_len = ParquetMetaDataReader::decode_footer(&footer).map_err(|e| { error::InvalidParquetSnafu { file: path, reason: format!("failed to decode footer, {e}"), @@ -118,14 +117,16 @@ impl<'a> MetadataLoader<'a> { if (metadata_len as usize) <= buffer_len - FOOTER_SIZE { // The whole metadata is in the first read let metadata_start = buffer_len - metadata_len as usize - FOOTER_SIZE; - let metadata = decode_metadata(&buffer[metadata_start..buffer_len - FOOTER_SIZE]) - .map_err(|e| { - error::InvalidParquetSnafu { - file: path, - reason: format!("failed to decode metadata, {e}"), - } - .build() - })?; + let metadata = ParquetMetaDataReader::decode_metadata( + &buffer[metadata_start..buffer_len - FOOTER_SIZE], + ) + .map_err(|e| { + error::InvalidParquetSnafu { + file: path, + reason: format!("failed to decode metadata, {e}"), + } + .build() + })?; Ok(metadata) } else { // The metadata is out of buffer, need to make a second read @@ -137,7 +138,7 @@ impl<'a> MetadataLoader<'a> { .context(error::OpenDalSnafu)? .to_vec(); - let metadata = decode_metadata(&data).map_err(|e| { + let metadata = ParquetMetaDataReader::decode_metadata(&data).map_err(|e| { error::InvalidParquetSnafu { file: path, reason: format!("failed to decode metadata, {e}"), diff --git a/src/mito2/src/sst/parquet/row_group.rs b/src/mito2/src/sst/parquet/row_group.rs index a3c92cb768..c3978f430b 100644 --- a/src/mito2/src/sst/parquet/row_group.rs +++ b/src/mito2/src/sst/parquet/row_group.rs @@ -24,10 +24,10 @@ use parquet::arrow::ProjectionMask; use parquet::column::page::{PageIterator, PageReader}; use parquet::errors::{ParquetError, Result}; use parquet::file::metadata::{ColumnChunkMetaData, ParquetMetaData, RowGroupMetaData}; +use parquet::file::page_index::offset_index::OffsetIndexMetaData; use parquet::file::properties::DEFAULT_PAGE_SIZE; use parquet::file::reader::{ChunkReader, Length}; use parquet::file::serialized_reader::SerializedPageReader; -use parquet::format::PageLocation; use store_api::storage::RegionId; use tokio::task::yield_now; @@ -39,31 +39,33 @@ use crate::sst::parquet::helper::fetch_byte_ranges; use crate::sst::parquet::page_reader::RowGroupCachedReader; pub(crate) struct RowGroupBase<'a> { - pub(crate) metadata: &'a RowGroupMetaData, - pub(crate) page_locations: Option<&'a [Vec]>, + metadata: &'a RowGroupMetaData, + pub(crate) offset_index: Option<&'a [OffsetIndexMetaData]>, /// Compressed page of each column. - pub(crate) column_chunks: Vec>>, + column_chunks: Vec>>, pub(crate) row_count: usize, /// Row group level cached pages for each column. /// /// These pages are uncompressed pages of a row group. /// `column_uncompressed_pages.len()` equals to `column_chunks.len()`. - pub(crate) column_uncompressed_pages: Vec>>, + column_uncompressed_pages: Vec>>, } impl<'a> RowGroupBase<'a> { pub(crate) fn new(parquet_meta: &'a ParquetMetaData, row_group_idx: usize) -> Self { let metadata = parquet_meta.row_group(row_group_idx); - // `page_locations` is always `None` if we don't set + // `offset_index` is always `None` if we don't set // [with_page_index()](https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderOptions.html#method.with_page_index) // to `true`. - let page_locations = parquet_meta + let offset_index = parquet_meta .offset_index() + // filter out empty offset indexes (old versions specified Some(vec![]) when no present) + .filter(|index| !index.is_empty()) .map(|x| x[row_group_idx].as_slice()); Self { metadata, - page_locations, + offset_index, column_chunks: vec![None; metadata.columns().len()], row_count: metadata.num_rows() as usize, column_uncompressed_pages: vec![None; metadata.columns().len()], @@ -73,7 +75,7 @@ impl<'a> RowGroupBase<'a> { pub(crate) fn calc_sparse_read_ranges( &self, projection: &ProjectionMask, - page_locations: &[Vec], + offset_index: &[OffsetIndexMetaData], selection: &RowSelection, ) -> (Vec>, Vec>) { // If we have a `RowSelection` and an `OffsetIndex` then only fetch pages required for the @@ -90,7 +92,7 @@ impl<'a> RowGroupBase<'a> { // then we need to also fetch a dictionary page. let mut ranges = vec![]; let (start, _len) = chunk_meta.byte_range(); - match page_locations[idx].first() { + match offset_index[idx].page_locations.first() { Some(first) if first.offset as u64 != start => { ranges.push(start..first.offset as u64); } @@ -99,7 +101,7 @@ impl<'a> RowGroupBase<'a> { ranges.extend( selection - .scan_ranges(&page_locations[idx]) + .scan_ranges(&offset_index[idx].page_locations) .iter() .map(|range| range.start as u64..range.end as u64), ); @@ -203,7 +205,11 @@ impl<'a> RowGroupBase<'a> { ))) } Some(data) => { - let page_locations = self.page_locations.map(|index| index[col_idx].clone()); + let page_locations = self + .offset_index + // filter out empty offset indexes (old versions specified Some(vec![]) when no present) + .filter(|index| !index.is_empty()) + .map(|index| index[col_idx].page_locations.clone()); SerializedPageReader::new( data.clone(), self.metadata.column(col_idx), @@ -245,13 +251,13 @@ impl<'a> InMemoryRowGroup<'a> { object_store: ObjectStore, ) -> Self { Self { - base: RowGroupBase::new(parquet_meta, row_group_idx), region_id, file_id, row_group_idx, cache_strategy, file_path, object_store, + base: RowGroupBase::new(parquet_meta, row_group_idx), } } @@ -261,10 +267,10 @@ impl<'a> InMemoryRowGroup<'a> { projection: &ProjectionMask, selection: Option<&RowSelection>, ) -> Result<()> { - if let Some((selection, page_locations)) = selection.zip(self.base.page_locations) { + if let Some((selection, offset_index)) = selection.zip(self.base.offset_index) { let (fetch_ranges, page_start_offsets) = self.base - .calc_sparse_read_ranges(projection, page_locations, selection); + .calc_sparse_read_ranges(projection, offset_index, selection); let chunk_data = self.fetch_bytes(&fetch_ranges).await?; // Assign sparse chunk data to base. diff --git a/src/operator/src/req_convert/common.rs b/src/operator/src/req_convert/common.rs index 518f2d55fa..700aa1384d 100644 --- a/src/operator/src/req_convert/common.rs +++ b/src/operator/src/req_convert/common.rs @@ -112,7 +112,7 @@ fn validate_rows(rows: &Option) -> Result<()> { for (col_idx, schema) in rows.schema.iter().enumerate() { let column_type = - ColumnDataTypeWrapper::try_new(schema.datatype, schema.datatype_extension.clone()) + ColumnDataTypeWrapper::try_new(schema.datatype, schema.datatype_extension) .context(ColumnDataTypeSnafu)? .into(); @@ -172,7 +172,7 @@ pub fn columns_to_rows(columns: Vec, row_count: u32) -> Result { column_name: column.column_name.clone(), datatype: column.datatype, semantic_type: column.semantic_type, - datatype_extension: column.datatype_extension.clone(), + datatype_extension: column.datatype_extension, options: column.options.clone(), }; schema.push(column_schema); diff --git a/src/operator/src/request.rs b/src/operator/src/request.rs index 0c1db682c3..36f368a681 100644 --- a/src/operator/src/request.rs +++ b/src/operator/src/request.rs @@ -109,7 +109,7 @@ impl Requester { .map(|partition| { RegionRequestBody::Compact(CompactRequest { region_id: partition.id.into(), - options: Some(request.compact_options.clone()), + options: Some(request.compact_options), }) }) .collect(); diff --git a/src/operator/src/statement/admin.rs b/src/operator/src/statement/admin.rs index 6c0413c2aa..885605e6d3 100644 --- a/src/operator/src/statement/admin.rs +++ b/src/operator/src/statement/admin.rs @@ -28,7 +28,7 @@ use datatypes::value::Value; use datatypes::vectors::VectorRef; use session::context::QueryContextRef; use snafu::{ensure, OptionExt, ResultExt}; -use sql::ast::{Expr, FunctionArg, FunctionArgExpr, Value as SqlValue}; +use sql::ast::{Expr, FunctionArg, FunctionArgExpr, FunctionArguments, Value as SqlValue}; use sql::statements::admin::Admin; use sql::statements::sql_value_to_value; @@ -53,7 +53,13 @@ impl StatementExecutor { .context(error::AdminFunctionNotFoundSnafu { name: func_name })?; let signature = admin_func.signature(); - let arg_values = func + let FunctionArguments::List(args) = &func.args else { + return error::BuildAdminFunctionArgsSnafu { + msg: format!("unsupported function args {}", func.args), + } + .fail(); + }; + let arg_values = args .args .iter() .map(|arg| { @@ -165,6 +171,8 @@ fn args_to_vector( } .fail() } + + TypeSignature::NullAry => Ok(vec![]), } } diff --git a/src/operator/src/statement/ddl.rs b/src/operator/src/statement/ddl.rs index 23cba11de8..e124faaf36 100644 --- a/src/operator/src/statement/ddl.rs +++ b/src/operator/src/statement/ddl.rs @@ -1477,7 +1477,7 @@ fn find_partition_entries( for column in column_defs { let column_name = &column.name; let data_type = ConcreteDataType::from( - ColumnDataTypeWrapper::try_new(column.data_type, column.datatype_extension.clone()) + ColumnDataTypeWrapper::try_new(column.data_type, column.datatype_extension) .context(ColumnDataTypeSnafu)?, ); column_name_and_type.insert(column_name, data_type); diff --git a/src/partition/src/manager.rs b/src/partition/src/manager.rs index c2e036f16a..1e33eda54d 100644 --- a/src/partition/src/manager.rs +++ b/src/partition/src/manager.rs @@ -327,7 +327,7 @@ fn create_partitions_from_region_routes( fn find_regions0(partition_rule: PartitionRuleRef, filter: &Expr) -> Result> { match filter { - Expr::BinaryExpr(BinaryExpr { left, op, right }) if op.is_comparison_operator() => { + Expr::BinaryExpr(BinaryExpr { left, op, right }) if op.supports_propagation() => { let column_op_value = match (left.as_ref(), right.as_ref()) { (Expr::Column(c), Expr::Literal(v)) => Some((&c.name, *op, v)), (Expr::Literal(v), Expr::Column(c)) => Some(( diff --git a/src/promql/src/extension_plan/empty_metric.rs b/src/promql/src/extension_plan/empty_metric.rs index 8478b52636..a1c8bc68e4 100644 --- a/src/promql/src/extension_plan/empty_metric.rs +++ b/src/promql/src/extension_plan/empty_metric.rs @@ -172,6 +172,25 @@ impl UserDefinedLogicalNodeCore for EmptyMetric { } } +impl PartialOrd for EmptyMetric { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding schema fields + match self.start.partial_cmp(&other.start) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.end.partial_cmp(&other.end) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.interval.partial_cmp(&other.interval) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + self.expr.partial_cmp(&other.expr) + } +} + #[derive(Debug, Clone)] pub struct EmptyMetricExec { start: Millisecond, @@ -247,6 +266,10 @@ impl ExecutionPlan for EmptyMetricExec { column_statistics: Statistics::unknown_column(&self.schema()), }) } + + fn name(&self) -> &str { + "EmptyMetricExec" + } } impl DisplayAs for EmptyMetricExec { diff --git a/src/promql/src/extension_plan/histogram_fold.rs b/src/promql/src/extension_plan/histogram_fold.rs index bfb745337a..8a03b46903 100644 --- a/src/promql/src/extension_plan/histogram_fold.rs +++ b/src/promql/src/extension_plan/histogram_fold.rs @@ -29,7 +29,7 @@ use datafusion::common::{ColumnStatistics, DFSchema, DFSchemaRef, Statistics}; use datafusion::error::{DataFusionError, Result as DataFusionResult}; use datafusion::execution::TaskContext; use datafusion::logical_expr::{LogicalPlan, UserDefinedLogicalNodeCore}; -use datafusion::physical_expr::{EquivalenceProperties, PhysicalSortRequirement}; +use datafusion::physical_expr::{EquivalenceProperties, LexRequirement, PhysicalSortRequirement}; use datafusion::physical_plan::expressions::{CastExpr as PhyCast, Column as PhyColumn}; use datafusion::physical_plan::metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricsSet}; use datafusion::physical_plan::{ @@ -218,6 +218,29 @@ impl HistogramFold { } } +impl PartialOrd for HistogramFold { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding output_schema + match self.le_column.partial_cmp(&other.le_column) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.ts_column.partial_cmp(&other.ts_column) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.input.partial_cmp(&other.input) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.field_column.partial_cmp(&other.field_column) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + self.quantile.partial_cmp(&other.quantile) + } +} + #[derive(Debug)] pub struct HistogramFoldExec { /// Index for `le` column in the schema of input. @@ -241,7 +264,7 @@ impl ExecutionPlan for HistogramFoldExec { &self.properties } - fn required_input_ordering(&self) -> Vec>> { + fn required_input_ordering(&self) -> Vec> { let mut cols = self .tag_col_exprs() .into_iter() @@ -274,7 +297,7 @@ impl ExecutionPlan for HistogramFoldExec { }), }); - vec![Some(cols)] + vec![Some(LexRequirement::new(cols))] } fn required_input_distribution(&self) -> Vec { @@ -352,10 +375,14 @@ impl ExecutionPlan for HistogramFoldExec { column_statistics: vec![ ColumnStatistics::new_unknown(); // plus one more for the removed column by function `convert_schema` - self.schema().all_fields().len() + 1 + self.schema().flattened_fields().len() + 1 ], }) } + + fn name(&self) -> &str { + "HistogramFoldExec" + } } impl HistogramFoldExec { diff --git a/src/promql/src/extension_plan/instant_manipulate.rs b/src/promql/src/extension_plan/instant_manipulate.rs index 1ab01f04aa..ec28f9a600 100644 --- a/src/promql/src/extension_plan/instant_manipulate.rs +++ b/src/promql/src/extension_plan/instant_manipulate.rs @@ -46,7 +46,7 @@ use crate::extension_plan::Millisecond; /// This plan will try to align the input time series, for every timestamp between /// `start` and `end` with step `interval`. Find in the `lookback` range if data /// is missing at the given timestamp. -#[derive(Debug, PartialEq, Eq, Hash)] +#[derive(Debug, PartialEq, Eq, Hash, PartialOrd)] pub struct InstantManipulate { start: Millisecond, end: Millisecond, @@ -290,10 +290,14 @@ impl ExecutionPlan for InstantManipulateExec { // TODO(ruihang): support this column statistics column_statistics: vec![ ColumnStatistics::new_unknown(); - self.schema().all_fields().len() + self.schema().flattened_fields().len() ], }) } + + fn name(&self) -> &str { + "InstantManipulateExec" + } } impl DisplayAs for InstantManipulateExec { diff --git a/src/promql/src/extension_plan/normalize.rs b/src/promql/src/extension_plan/normalize.rs index 5abaad427c..96be166801 100644 --- a/src/promql/src/extension_plan/normalize.rs +++ b/src/promql/src/extension_plan/normalize.rs @@ -47,7 +47,7 @@ use crate::extension_plan::Millisecond; /// - bias sample's timestamp by offset /// - sort the record batch based on timestamp column /// - remove NaN values (optional) -#[derive(Debug, PartialEq, Eq, Hash)] +#[derive(Debug, PartialEq, Eq, Hash, PartialOrd)] pub struct SeriesNormalize { offset: Millisecond, time_index_column_name: String, @@ -229,6 +229,10 @@ impl ExecutionPlan for SeriesNormalizeExec { fn statistics(&self) -> DataFusionResult { self.input.statistics() } + + fn name(&self) -> &str { + "SeriesNormalizeExec" + } } impl DisplayAs for SeriesNormalizeExec { diff --git a/src/promql/src/extension_plan/range_manipulate.rs b/src/promql/src/extension_plan/range_manipulate.rs index 1aef1e8343..c2c4f9b42f 100644 --- a/src/promql/src/extension_plan/range_manipulate.rs +++ b/src/promql/src/extension_plan/range_manipulate.rs @@ -206,6 +206,37 @@ impl RangeManipulate { } } +impl PartialOrd for RangeManipulate { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding output_schema + match self.start.partial_cmp(&other.start) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.end.partial_cmp(&other.end) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.interval.partial_cmp(&other.interval) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.range.partial_cmp(&other.range) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.time_index.partial_cmp(&other.time_index) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.field_columns.partial_cmp(&other.field_columns) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + self.input.partial_cmp(&other.input) + } +} + impl UserDefinedLogicalNodeCore for RangeManipulate { fn name(&self) -> &str { Self::name() @@ -385,6 +416,10 @@ impl ExecutionPlan for RangeManipulateExec { column_statistics: Statistics::unknown_column(&self.schema()), }) } + + fn name(&self) -> &str { + "RangeManipulateExec" + } } impl DisplayAs for RangeManipulateExec { diff --git a/src/promql/src/extension_plan/scalar_calculate.rs b/src/promql/src/extension_plan/scalar_calculate.rs index b206c69b78..7cff38d73f 100644 --- a/src/promql/src/extension_plan/scalar_calculate.rs +++ b/src/promql/src/extension_plan/scalar_calculate.rs @@ -195,6 +195,37 @@ impl ScalarCalculate { } } +impl PartialOrd for ScalarCalculate { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding output_schema + match self.start.partial_cmp(&other.start) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.end.partial_cmp(&other.end) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.interval.partial_cmp(&other.interval) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.time_index.partial_cmp(&other.time_index) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.tag_columns.partial_cmp(&other.tag_columns) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.field_column.partial_cmp(&other.field_column) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + self.input.partial_cmp(&other.input) + } +} + impl UserDefinedLogicalNodeCore for ScalarCalculate { fn name(&self) -> &str { Self::name() @@ -353,6 +384,10 @@ impl ExecutionPlan for ScalarCalculateExec { column_statistics: Statistics::unknown_column(&self.schema()), }) } + + fn name(&self) -> &str { + "ScalarCalculateExec" + } } impl DisplayAs for ScalarCalculateExec { diff --git a/src/promql/src/extension_plan/series_divide.rs b/src/promql/src/extension_plan/series_divide.rs index b0ce0219ea..fbdaa7dec3 100644 --- a/src/promql/src/extension_plan/series_divide.rs +++ b/src/promql/src/extension_plan/series_divide.rs @@ -24,7 +24,7 @@ use datafusion::common::{DFSchema, DFSchemaRef}; use datafusion::error::Result as DataFusionResult; use datafusion::execution::context::TaskContext; use datafusion::logical_expr::{EmptyRelation, Expr, LogicalPlan, UserDefinedLogicalNodeCore}; -use datafusion::physical_expr::PhysicalSortRequirement; +use datafusion::physical_expr::{LexRequirement, PhysicalSortRequirement}; use datafusion::physical_plan::expressions::Column as ColumnExpr; use datafusion::physical_plan::metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricsSet}; use datafusion::physical_plan::{ @@ -40,7 +40,7 @@ use snafu::ResultExt; use crate::error::{DeserializeSnafu, Result}; use crate::metrics::PROMQL_SERIES_COUNT; -#[derive(Debug, PartialEq, Eq, Hash)] +#[derive(Debug, PartialEq, Eq, Hash, PartialOrd)] pub struct SeriesDivide { tag_columns: Vec, input: LogicalPlan, @@ -146,7 +146,7 @@ impl ExecutionPlan for SeriesDivideExec { vec![Distribution::SinglePartition] } - fn required_input_ordering(&self) -> Vec>> { + fn required_input_ordering(&self) -> Vec> { let input_schema = self.input.schema(); let exprs: Vec = self .tag_columns @@ -158,7 +158,7 @@ impl ExecutionPlan for SeriesDivideExec { }) .collect(); if !exprs.is_empty() { - vec![Some(exprs)] + vec![Some(LexRequirement::new(exprs))] } else { vec![None] } @@ -217,6 +217,10 @@ impl ExecutionPlan for SeriesDivideExec { fn metrics(&self) -> Option { Some(self.metric.clone_inner()) } + + fn name(&self) -> &str { + "SeriesDivideExec" + } } impl DisplayAs for SeriesDivideExec { diff --git a/src/promql/src/extension_plan/union_distinct_on.rs b/src/promql/src/extension_plan/union_distinct_on.rs index 2da0fd54fa..83af93edc9 100644 --- a/src/promql/src/extension_plan/union_distinct_on.rs +++ b/src/promql/src/extension_plan/union_distinct_on.rs @@ -110,6 +110,25 @@ impl UnionDistinctOn { } } +impl PartialOrd for UnionDistinctOn { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding output_schema + match self.left.partial_cmp(&other.left) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.right.partial_cmp(&other.right) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + match self.compare_keys.partial_cmp(&other.compare_keys) { + Some(core::cmp::Ordering::Equal) => {} + ord => return ord, + } + self.ts_col.partial_cmp(&other.ts_col) + } +} + impl UserDefinedLogicalNodeCore for UnionDistinctOn { fn name(&self) -> &str { Self::name() @@ -262,6 +281,10 @@ impl ExecutionPlan for UnionDistinctOnExec { fn metrics(&self) -> Option { Some(self.metric.clone_inner()) } + + fn name(&self) -> &str { + "UnionDistinctOnExec" + } } impl DisplayAs for UnionDistinctOnExec { diff --git a/src/promql/src/functions/extrapolate_rate.rs b/src/promql/src/functions/extrapolate_rate.rs index 27ac86d8af..6ced7c9d4d 100644 --- a/src/promql/src/functions/extrapolate_rate.rs +++ b/src/promql/src/functions/extrapolate_rate.rs @@ -76,7 +76,7 @@ impl ExtrapolatedRate IDelta { create_udf( Self::name(), Self::input_type(), - Arc::new(Self::return_type()), + Self::return_type(), Volatility::Immutable, Arc::new(Self::calc) as _, ) diff --git a/src/promql/src/functions/predict_linear.rs b/src/promql/src/functions/predict_linear.rs index 647205f979..b64e1c1066 100644 --- a/src/promql/src/functions/predict_linear.rs +++ b/src/promql/src/functions/predict_linear.rs @@ -54,7 +54,7 @@ impl PredictLinear { create_udf( Self::name(), input_types, - Arc::new(DataType::Float64), + DataType::Float64, Volatility::Immutable, Arc::new(move |input: &_| Self::new(t).predict_linear(input)) as _, ) diff --git a/src/promql/src/functions/quantile.rs b/src/promql/src/functions/quantile.rs index 3721e0da8d..19f346ee18 100644 --- a/src/promql/src/functions/quantile.rs +++ b/src/promql/src/functions/quantile.rs @@ -50,7 +50,7 @@ impl QuantileOverTime { create_udf( Self::name(), input_types, - Arc::new(DataType::Float64), + DataType::Float64, Volatility::Immutable, Arc::new(move |input: &_| Self::new(quantile).quantile_over_time(input)) as _, ) diff --git a/src/promql/src/functions/test_util.rs b/src/promql/src/functions/test_util.rs index 8f9558e590..a4e3f16ee3 100644 --- a/src/promql/src/functions/test_util.rs +++ b/src/promql/src/functions/test_util.rs @@ -28,17 +28,19 @@ pub fn simple_range_udf_runner( input_value: RangeArray, expected: Vec>, ) { + let num_rows = input_ts.len(); let input = vec![ ColumnarValue::Array(Arc::new(input_ts.into_dict())), ColumnarValue::Array(Arc::new(input_value.into_dict())), ]; - let eval_result: Vec> = extract_array(&(range_fn.fun())(&input).unwrap()) - .unwrap() - .as_any() - .downcast_ref::() - .unwrap() - .iter() - .collect(); + let eval_result: Vec> = + extract_array(&range_fn.invoke_batch(&input, num_rows).unwrap()) + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .iter() + .collect(); assert_eq!(eval_result.len(), expected.len()); assert!(eval_result .iter() diff --git a/src/query/src/datafusion.rs b/src/query/src/datafusion.rs index f295a2c9b3..4bb52de316 100644 --- a/src/query/src/datafusion.rs +++ b/src/query/src/datafusion.rs @@ -50,9 +50,9 @@ use crate::dataframe::DataFrame; pub use crate::datafusion::planner::DfContextProviderAdapter; use crate::dist_plan::MergeScanLogicalPlan; use crate::error::{ - CatalogSnafu, ConvertSchemaSnafu, CreateRecordBatchSnafu, DataFusionSnafu, - MissingTableMutationHandlerSnafu, MissingTimestampColumnSnafu, QueryExecutionSnafu, Result, - TableMutationSnafu, TableNotFoundSnafu, TableReadOnlySnafu, UnsupportedExprSnafu, + CatalogSnafu, ConvertSchemaSnafu, CreateRecordBatchSnafu, MissingTableMutationHandlerSnafu, + MissingTimestampColumnSnafu, QueryExecutionSnafu, Result, TableMutationSnafu, + TableNotFoundSnafu, TableReadOnlySnafu, UnsupportedExprSnafu, }; use crate::executor::QueryExecutor; use crate::metrics::{OnDone, QUERY_STAGE_ELAPSED}; @@ -102,7 +102,7 @@ impl DatafusionQueryEngine { query_ctx: QueryContextRef, ) -> Result { ensure!( - matches!(dml.op, WriteOp::InsertInto | WriteOp::Delete), + matches!(dml.op, WriteOp::Insert(_) | WriteOp::Delete), UnsupportedExprSnafu { name: format!("DML op {}", dml.op), } @@ -137,7 +137,8 @@ impl DatafusionQueryEngine { .context(QueryExecutionSnafu)?; match dml.op { - WriteOp::InsertInto => { + WriteOp::Insert(_) => { + // We ignore the insert op. let output = self .insert(&table_name, column_vectors, query_ctx.clone()) .await?; @@ -283,6 +284,7 @@ impl DatafusionQueryEngine { .context(error::DatafusionSnafu) .map_err(BoxedError::new) .context(QueryExecutionSnafu)?; + // skip optimize for MergeScan let optimized_plan = if let DfLogicalPlan::Extension(ext) = &analyzed_plan && ext.node.name() == MergeScanLogicalPlan::name() @@ -339,31 +341,36 @@ impl DatafusionQueryEngine { #[tracing::instrument(skip_all)] fn optimize_physical_plan( &self, - ctx: &mut QueryEngineContext, + _ctx: &mut QueryEngineContext, plan: Arc, ) -> Result> { let _timer = metrics::OPTIMIZE_PHYSICAL_ELAPSED.start_timer(); - let state = ctx.state(); - let config = state.config_options(); + // TODO(ruihang): `self.create_physical_plan()` already optimize the plan, check + // if we need to optimize it again here. + // let state = ctx.state(); + // let config = state.config_options(); + // skip optimize AnalyzeExec plan let optimized_plan = if let Some(analyze_plan) = plan.as_any().downcast_ref::() { - let mut new_plan = analyze_plan.input().clone(); - for optimizer in state.physical_optimizers() { - new_plan = optimizer - .optimize(new_plan, config) - .context(DataFusionSnafu)?; - } - Arc::new(DistAnalyzeExec::new(new_plan)) + Arc::new(DistAnalyzeExec::new(analyze_plan.input().clone())) + // let mut new_plan = analyze_plan.input().clone(); + // for optimizer in state.physical_optimizers() { + // new_plan = optimizer + // .optimize(new_plan, config) + // .context(DataFusionSnafu)?; + // } + // Arc::new(DistAnalyzeExec::new(new_plan)) } else { - let mut new_plan = plan; - for optimizer in state.physical_optimizers() { - new_plan = optimizer - .optimize(new_plan, config) - .context(DataFusionSnafu)?; - } - new_plan + plan + // let mut new_plan = plan; + // for optimizer in state.physical_optimizers() { + // new_plan = optimizer + // .optimize(new_plan, config) + // .context(DataFusionSnafu)?; + // } + // new_plan }; Ok(optimized_plan) @@ -574,12 +581,11 @@ mod tests { .await .unwrap(); - // TODO(sunng87): do not rely on to_string for compare assert_eq!( - format!("{plan:?}"), + plan.to_string(), r#"Limit: skip=0, fetch=20 - Projection: SUM(numbers.number) - Aggregate: groupBy=[[]], aggr=[[SUM(numbers.number)]] + Projection: sum(numbers.number) + Aggregate: groupBy=[[]], aggr=[[sum(numbers.number)]] TableScan: numbers"# ); } @@ -605,7 +611,7 @@ mod tests { assert_eq!(numbers[0].num_columns(), 1); assert_eq!(1, numbers[0].schema.num_columns()); assert_eq!( - "SUM(numbers.number)", + "sum(numbers.number)", numbers[0].schema.column_schemas()[0].name ); @@ -683,11 +689,11 @@ mod tests { assert_eq!( schema.column_schemas()[0], ColumnSchema::new( - "SUM(numbers.number)", + "sum(numbers.number)", ConcreteDataType::uint64_datatype(), true ) ); - assert_eq!("Limit: skip=0, fetch=20\n Aggregate: groupBy=[[]], aggr=[[SUM(CAST(numbers.number AS UInt64))]]\n TableScan: numbers projection=[number]", format!("{}", logical_plan.display_indent())); + assert_eq!("Limit: skip=0, fetch=20\n Aggregate: groupBy=[[]], aggr=[[sum(CAST(numbers.number AS UInt64))]]\n TableScan: numbers projection=[number]", format!("{}", logical_plan.display_indent())); } } diff --git a/src/query/src/datafusion/planner.rs b/src/query/src/datafusion/planner.rs index 7927ba1685..a82cc9b570 100644 --- a/src/query/src/datafusion/planner.rs +++ b/src/query/src/datafusion/planner.rs @@ -20,12 +20,19 @@ use arrow_schema::DataType; use catalog::table_source::DfTableSourceProvider; use common_function::scalars::udf::create_udf; use common_query::logical_plan::create_aggregate_function; -use datafusion::catalog::TableReference; +use datafusion::common::TableReference; +use datafusion::datasource::cte_worktable::CteWorkTable; +use datafusion::datasource::file_format::{format_as_file_type, FileFormatFactory}; +use datafusion::datasource::provider_as_source; use datafusion::error::Result as DfResult; use datafusion::execution::context::SessionState; +use datafusion::execution::SessionStateDefaults; use datafusion::sql::planner::ContextProvider; use datafusion::variable::VarType; use datafusion_common::config::ConfigOptions; +use datafusion_common::file_options::file_type::FileType; +use datafusion_common::DataFusionError; +use datafusion_expr::planner::{ExprPlanner, TypePlanner}; use datafusion_expr::var_provider::is_system_variables; use datafusion_expr::{AggregateUDF, ScalarUDF, TableSource, WindowUDF}; use datafusion_sql::parser::Statement as DfStatement; @@ -41,6 +48,14 @@ pub struct DfContextProviderAdapter { tables: HashMap>, table_provider: DfTableSourceProvider, query_ctx: QueryContextRef, + + // Fields from session state defaults: + /// Holds registered external FileFormat implementations + /// DataFusion doesn't pub this field, so we need to store it here. + file_formats: HashMap>, + /// Provides support for customising the SQL planner, e.g. to add support for custom operators like `->>` or `?` + /// DataFusion doesn't pub this field, so we need to store it here. + expr_planners: Vec>, } impl DfContextProviderAdapter { @@ -70,12 +85,19 @@ impl DfContextProviderAdapter { ); let tables = resolve_tables(table_names, &mut table_provider).await?; + let file_formats = SessionStateDefaults::default_file_formats() + .into_iter() + .map(|format| (format.get_ext().to_lowercase(), format)) + .collect(); + Ok(Self { engine_state, session_state, tables, table_provider, query_ctx, + file_formats, + expr_planners: SessionStateDefaults::default_expr_planners(), }) } } @@ -153,8 +175,8 @@ impl ContextProvider for DfContextProviderAdapter { ) } - fn get_window_meta(&self, _name: &str) -> Option> { - None + fn get_window_meta(&self, name: &str) -> Option> { + self.session_state.window_functions().get(name).cloned() } fn get_variable_type(&self, variable_names: &[String]) -> Option { @@ -180,17 +202,64 @@ impl ContextProvider for DfContextProviderAdapter { } fn udf_names(&self) -> Vec { - // TODO(LFC): Impl it. - vec![] + let mut names = self.engine_state.udf_names(); + names.extend(self.session_state.scalar_functions().keys().cloned()); + names } fn udaf_names(&self) -> Vec { - // TODO(LFC): Impl it. - vec![] + let mut names = self.engine_state.udaf_names(); + names.extend(self.session_state.aggregate_functions().keys().cloned()); + names } fn udwf_names(&self) -> Vec { - // TODO(LFC): Impl it. - vec![] + self.session_state + .window_functions() + .keys() + .cloned() + .collect() + } + + fn get_file_type(&self, ext: &str) -> DfResult> { + self.file_formats + .get(&ext.to_lowercase()) + .ok_or_else(|| { + DataFusionError::Plan(format!("There is no registered file format with ext {ext}")) + }) + .map(|file_type| format_as_file_type(Arc::clone(file_type))) + } + + fn get_table_function_source( + &self, + name: &str, + args: Vec, + ) -> DfResult> { + let tbl_func = self + .session_state + .table_functions() + .get(name) + .cloned() + .ok_or_else(|| DataFusionError::Plan(format!("table function '{name}' not found")))?; + let provider = tbl_func.create_table_provider(&args)?; + + Ok(provider_as_source(provider)) + } + + fn create_cte_work_table( + &self, + name: &str, + schema: arrow_schema::SchemaRef, + ) -> DfResult> { + let table = Arc::new(CteWorkTable::new(name, schema)); + Ok(provider_as_source(table)) + } + + fn get_expr_planners(&self) -> &[Arc] { + &self.expr_planners + } + + fn get_type_planner(&self) -> Option> { + None } } diff --git a/src/query/src/dist_plan/analyzer.rs b/src/query/src/dist_plan/analyzer.rs index 335c23c58a..88ee9b2c57 100644 --- a/src/query/src/dist_plan/analyzer.rs +++ b/src/query/src/dist_plan/analyzer.rs @@ -33,8 +33,10 @@ use crate::dist_plan::commutativity::{ partial_commutative_transformer, Categorizer, Commutativity, }; use crate::dist_plan::merge_scan::MergeScanLogicalPlan; +use crate::plan::ExtractExpr; use crate::query_engine::DefaultSerializer; +#[derive(Debug)] pub struct DistPlannerAnalyzer; impl AnalyzerRule for DistPlannerAnalyzer { @@ -64,7 +66,7 @@ impl AnalyzerRule for DistPlannerAnalyzer { impl DistPlannerAnalyzer { fn inspect_plan_with_subquery(plan: LogicalPlan) -> DfResult> { let exprs = plan - .expressions() + .expressions_consider_join() .into_iter() .map(|e| e.transform(&Self::transform_subquery).map(|x| x.data)) .collect::>>()?; @@ -267,7 +269,8 @@ impl PlanRewriter { // expand stages for new_stage in self.stage.drain(..) { - node = new_stage.with_new_exprs(new_stage.expressions(), vec![node.clone()])?; + node = new_stage + .with_new_exprs(new_stage.expressions_consider_join(), vec![node.clone()])?; } self.set_expanded(); @@ -376,8 +379,9 @@ mod test { use std::sync::Arc; use datafusion::datasource::DefaultTableSource; + use datafusion::functions_aggregate::expr_fn::avg; use datafusion_common::JoinType; - use datafusion_expr::{avg, col, lit, Expr, LogicalPlanBuilder}; + use datafusion_expr::{col, lit, Expr, LogicalPlanBuilder}; use table::table::adapter::DfTableProviderAdapter; use table::table::numbers::NumbersTable; @@ -413,7 +417,7 @@ mod test { " TableScan: t", ] .join("\n"); - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } #[test] @@ -433,7 +437,7 @@ mod test { let config = ConfigOptions::default(); let result = DistPlannerAnalyzer {}.analyze(plan, &config).unwrap(); let expected = "MergeScan [is_placeholder=false]"; - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } #[test] @@ -460,7 +464,7 @@ mod test { " MergeScan [is_placeholder=false]", ] .join("\n"); - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } #[test] @@ -480,7 +484,7 @@ mod test { let config = ConfigOptions::default(); let result = DistPlannerAnalyzer {}.analyze(plan, &config).unwrap(); let expected = "MergeScan [is_placeholder=false]"; - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } #[test] @@ -503,7 +507,11 @@ mod test { let plan = LogicalPlanBuilder::scan_with_filters("t", left_source, None, vec![]) .unwrap() - .join_using(right_plan, JoinType::LeftSemi, vec!["number"]) + .join_on( + right_plan, + JoinType::LeftSemi, + vec![col("t.number").eq(col("right.number"))], + ) .unwrap() .limit(0, Some(1)) .unwrap() @@ -513,10 +521,10 @@ mod test { let config = ConfigOptions::default(); let result = DistPlannerAnalyzer {}.analyze(plan, &config).unwrap(); let expected = "Limit: skip=0, fetch=1\ - \n LeftSemi Join: Using t.number = right.number\ + \n LeftSemi Join: Filter: t.number = right.number\ \n MergeScan [is_placeholder=false]\ \n SubqueryAlias: right\ \n MergeScan [is_placeholder=false]"; - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } } diff --git a/src/query/src/dist_plan/commutativity.rs b/src/query/src/dist_plan/commutativity.rs index 45378e532c..4f96966b46 100644 --- a/src/query/src/dist_plan/commutativity.rs +++ b/src/query/src/dist_plan/commutativity.rs @@ -15,7 +15,6 @@ use std::collections::HashSet; use std::sync::Arc; -use datafusion_expr::utils::exprlist_to_columns; use datafusion_expr::{Expr, LogicalPlan, UserDefinedLogicalNode}; use promql::extension_plan::{ EmptyMetric, InstantManipulate, RangeManipulate, SeriesDivide, SeriesNormalize, @@ -74,7 +73,6 @@ impl Categorizer { Commutativity::ConditionalCommutative(Some(Arc::new(merge_sort_transformer))) } LogicalPlan::Join(_) => Commutativity::NonCommutative, - LogicalPlan::CrossJoin(_) => Commutativity::NonCommutative, LogicalPlan::Repartition(_) => { // unsupported? or non-commutative Commutativity::Unimplemented @@ -89,7 +87,7 @@ impl Categorizer { // wait for https://github.com/apache/arrow-datafusion/pull/7669 if partition_cols.is_empty() && limit.fetch.is_some() { Commutativity::Commutative - } else if limit.skip == 0 && limit.fetch.is_some() { + } else if limit.skip.is_none() && limit.fetch.is_some() { Commutativity::PartialCommutative } else { Commutativity::Unimplemented @@ -104,7 +102,6 @@ impl Categorizer { LogicalPlan::Values(_) => Commutativity::Unsupported, LogicalPlan::Explain(_) => Commutativity::Unsupported, LogicalPlan::Analyze(_) => Commutativity::Unsupported, - LogicalPlan::Prepare(_) => Commutativity::Unsupported, LogicalPlan::DescribeTable(_) => Commutativity::Unsupported, LogicalPlan::Dml(_) => Commutativity::Unsupported, LogicalPlan::Ddl(_) => Commutativity::Unsupported, @@ -144,7 +141,6 @@ impl Categorizer { | Expr::IsNotFalse(_) | Expr::Negative(_) | Expr::Between(_) - | Expr::Sort(_) | Expr::Exists(_) | Expr::InList(_) | Expr::ScalarFunction(_) => Commutativity::Commutative, @@ -174,8 +170,8 @@ impl Categorizer { /// In this case the plan can be treated as fully commutative. fn check_partition(exprs: &[Expr], partition_cols: &[String]) -> bool { let mut ref_cols = HashSet::new(); - if exprlist_to_columns(exprs, &mut ref_cols).is_err() { - return false; + for expr in exprs { + expr.add_column_refs(&mut ref_cols); } let ref_cols = ref_cols .into_iter() diff --git a/src/query/src/dist_plan/merge_scan.rs b/src/query/src/dist_plan/merge_scan.rs index 8ed77eabd4..47a9512583 100644 --- a/src/query/src/dist_plan/merge_scan.rs +++ b/src/query/src/dist_plan/merge_scan.rs @@ -53,7 +53,7 @@ use crate::error::ConvertSchemaSnafu; use crate::metrics::{MERGE_SCAN_ERRORS_TOTAL, MERGE_SCAN_POLL_ELAPSED, MERGE_SCAN_REGIONS}; use crate::region_query::RegionQueryHandlerRef; -#[derive(Debug, Hash, PartialEq, Eq, Clone)] +#[derive(Debug, Hash, PartialOrd, PartialEq, Eq, Clone)] pub struct MergeScanLogicalPlan { /// In logical plan phase it only contains one input input: LogicalPlan, @@ -347,6 +347,10 @@ impl ExecutionPlan for MergeScanExec { fn metrics(&self) -> Option { Some(self.metric.clone_inner()) } + + fn name(&self) -> &str { + "MergeScanExec" + } } impl DisplayAs for MergeScanExec { diff --git a/src/query/src/dist_plan/merge_sort.rs b/src/query/src/dist_plan/merge_sort.rs index df0a9fda10..d9bc6119c0 100644 --- a/src/query/src/dist_plan/merge_sort.rs +++ b/src/query/src/dist_plan/merge_sort.rs @@ -20,14 +20,14 @@ use std::fmt; use std::sync::Arc; use datafusion_common::{DataFusionError, Result}; -use datafusion_expr::{Expr, Extension, LogicalPlan, UserDefinedLogicalNodeCore}; +use datafusion_expr::{Extension, LogicalPlan, SortExpr, UserDefinedLogicalNodeCore}; /// MergeSort Logical Plan, have same field as `Sort`, but indicate it is a merge sort, /// which assume each input partition is a sorted stream, and will use `SortPreserveingMergeExec` /// to merge them into a single sorted stream. -#[derive(Hash, PartialEq, Eq, Clone)] +#[derive(Hash, PartialOrd, PartialEq, Eq, Clone)] pub struct MergeSortLogicalPlan { - pub expr: Vec, + pub expr: Vec, pub input: Arc, pub fetch: Option, } @@ -39,7 +39,7 @@ impl fmt::Debug for MergeSortLogicalPlan { } impl MergeSortLogicalPlan { - pub fn new(input: Arc, expr: Vec, fetch: Option) -> Self { + pub fn new(input: Arc, expr: Vec, fetch: Option) -> Self { Self { input, expr, fetch } } @@ -80,7 +80,7 @@ impl UserDefinedLogicalNodeCore for MergeSortLogicalPlan { // Allow further optimization fn expressions(&self) -> Vec { - self.expr.clone() + self.expr.iter().map(|sort| sort.expr.clone()).collect() } fn fmt_for_explain(&self, f: &mut fmt::Formatter) -> fmt::Result { @@ -103,7 +103,12 @@ impl UserDefinedLogicalNodeCore for MergeSortLogicalPlan { mut inputs: Vec, ) -> Result { let mut zelf = self.clone(); - zelf.expr = exprs; + zelf.expr = zelf + .expr + .into_iter() + .zip(exprs) + .map(|(sort, expr)| sort.with_expr(expr)) + .collect(); zelf.input = Arc::new(inputs.pop().ok_or_else(|| { DataFusionError::Internal("Expected exactly one input with MergeSort".to_string()) })?); diff --git a/src/query/src/dummy_catalog.rs b/src/query/src/dummy_catalog.rs index c4ae1de03a..ae57757683 100644 --- a/src/query/src/dummy_catalog.rs +++ b/src/query/src/dummy_catalog.rs @@ -15,16 +15,15 @@ //! Dummy catalog for region server. use std::any::Any; +use std::fmt; use std::sync::{Arc, Mutex}; use api::v1::SemanticType; use async_trait::async_trait; use common_recordbatch::filter::SimpleFilterEvaluator; use common_recordbatch::OrderOption; -use datafusion::catalog::schema::SchemaProvider; -use datafusion::catalog::{CatalogProvider, CatalogProviderList}; +use datafusion::catalog::{CatalogProvider, CatalogProviderList, SchemaProvider, Session}; use datafusion::datasource::TableProvider; -use datafusion::execution::context::SessionState; use datafusion::physical_plan::ExecutionPlan; use datafusion_common::DataFusionError; use datafusion_expr::{Expr, TableProviderFilterPushDown, TableType}; @@ -38,7 +37,7 @@ use table::table::scan::RegionScanExec; use crate::error::{GetRegionMetadataSnafu, Result}; /// Resolve to the given region (specified by [RegionId]) unconditionally. -#[derive(Clone)] +#[derive(Clone, Debug)] pub struct DummyCatalogList { catalog: DummyCatalogProvider, } @@ -81,7 +80,7 @@ impl CatalogProviderList for DummyCatalogList { } /// A dummy catalog provider for [DummyCatalogList]. -#[derive(Clone)] +#[derive(Clone, Debug)] struct DummyCatalogProvider { schema: DummySchemaProvider, } @@ -101,7 +100,7 @@ impl CatalogProvider for DummyCatalogProvider { } /// A dummy schema provider for [DummyCatalogList]. -#[derive(Clone)] +#[derive(Clone, Debug)] struct DummySchemaProvider { table: Arc, } @@ -138,6 +137,16 @@ pub struct DummyTableProvider { scan_request: Arc>, } +impl fmt::Debug for DummyTableProvider { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("DummyTableProvider") + .field("region_id", &self.region_id) + .field("metadata", &self.metadata) + .field("scan_request", &self.scan_request) + .finish() + } +} + #[async_trait] impl TableProvider for DummyTableProvider { fn as_any(&self) -> &dyn Any { @@ -154,16 +163,13 @@ impl TableProvider for DummyTableProvider { async fn scan( &self, - _state: &SessionState, + _state: &dyn Session, projection: Option<&Vec>, filters: &[Expr], limit: Option, ) -> datafusion::error::Result> { let mut request = self.scan_request.lock().unwrap().clone(); - request.projection = match projection { - Some(x) if !x.is_empty() => Some(x.clone()), - _ => None, - }; + request.projection = projection.cloned(); request.filters = filters.to_vec(); request.limit = limit; diff --git a/src/query/src/optimizer/count_wildcard.rs b/src/query/src/optimizer/count_wildcard.rs index ccea20cb91..9621a8e159 100644 --- a/src/query/src/optimizer/count_wildcard.rs +++ b/src/query/src/optimizer/count_wildcard.rs @@ -17,7 +17,7 @@ use datafusion_common::tree_node::{ Transformed, TransformedResult, TreeNode, TreeNodeRecursion, TreeNodeVisitor, }; use datafusion_common::{Column, Result as DataFusionResult}; -use datafusion_expr::expr::{AggregateFunction, AggregateFunctionDefinition, WindowFunction}; +use datafusion_expr::expr::{AggregateFunction, WindowFunction}; use datafusion_expr::utils::COUNT_STAR_EXPANSION; use datafusion_expr::{col, lit, Expr, LogicalPlan, WindowFunctionDefinition}; use datafusion_optimizer::utils::NamePreserver; @@ -30,6 +30,7 @@ use table::table::adapter::DfTableProviderAdapter; /// faster to read comparing to PRIMARY KEYs. /// /// [`CountWildcardRule`]: datafusion::optimizer::analyzer::CountWildcardRule +#[derive(Debug)] pub struct CountWildcardToTimeIndexRule; impl AnalyzerRule for CountWildcardToTimeIndexRule { @@ -56,7 +57,7 @@ impl CountWildcardToTimeIndexRule { vec![lit(COUNT_STAR_EXPANSION)] }; plan.map_expressions(|expr| { - let original_name = name_preserver.save(&expr)?; + let original_name = name_preserver.save(&expr); let transformed_expr = expr.transform_up(|expr| match expr { Expr::WindowFunction(mut window_function) if Self::is_count_star_window_aggregate(&window_function) => @@ -74,7 +75,7 @@ impl CountWildcardToTimeIndexRule { } _ => Ok(Transformed::no(expr)), })?; - transformed_expr.map_data(|data| original_name.restore(data)) + Ok(transformed_expr.update_data(|data| original_name.restore(data))) }) } @@ -105,27 +106,23 @@ impl CountWildcardToTimeIndexRule { /// Utility functions from the original rule. impl CountWildcardToTimeIndexRule { fn is_wildcard(expr: &Expr) -> bool { - matches!(expr, Expr::Wildcard { qualifier: None }) + matches!(expr, Expr::Wildcard { .. }) } fn is_count_star_aggregate(aggregate_function: &AggregateFunction) -> bool { - matches!( - &aggregate_function.func_def, - AggregateFunctionDefinition::BuiltIn( - datafusion_expr::aggregate_function::AggregateFunction::Count, - ) - ) && aggregate_function.args.len() == 1 - && Self::is_wildcard(&aggregate_function.args[0]) + matches!(aggregate_function, + AggregateFunction { + func, + args, + .. + } if func.name() == "count" && (args.len() == 1 && Self::is_wildcard(&args[0]) || args.is_empty())) } fn is_count_star_window_aggregate(window_function: &WindowFunction) -> bool { - matches!( - &window_function.fun, - WindowFunctionDefinition::AggregateFunction( - datafusion_expr::aggregate_function::AggregateFunction::Count, - ) - ) && window_function.args.len() == 1 - && Self::is_wildcard(&window_function.args[0]) + let args = &window_function.args; + matches!(window_function.fun, + WindowFunctionDefinition::AggregateUDF(ref udaf) + if udaf.name() == "count" && (args.len() == 1 && Self::is_wildcard(&args[0]) || args.is_empty())) } } @@ -187,7 +184,8 @@ impl TimeIndexFinder { mod test { use std::sync::Arc; - use datafusion_expr::{count, wildcard, LogicalPlanBuilder}; + use datafusion::functions_aggregate::count::count; + use datafusion_expr::{wildcard, LogicalPlanBuilder}; use table::table::numbers::NumbersTable; use super::*; diff --git a/src/query/src/optimizer/parallelize_scan.rs b/src/query/src/optimizer/parallelize_scan.rs index a9e0a33024..eb712ec920 100644 --- a/src/query/src/optimizer/parallelize_scan.rs +++ b/src/query/src/optimizer/parallelize_scan.rs @@ -25,6 +25,7 @@ use datafusion_common::{DataFusionError, Result}; use store_api::region_engine::PartitionRange; use table::table::scan::RegionScanExec; +#[derive(Debug)] pub struct ParallelizeScan; impl PhysicalOptimizerRule for ParallelizeScan { diff --git a/src/query/src/optimizer/remove_duplicate.rs b/src/query/src/optimizer/remove_duplicate.rs index 7f3ba29668..77c0ddd788 100644 --- a/src/query/src/optimizer/remove_duplicate.rs +++ b/src/query/src/optimizer/remove_duplicate.rs @@ -27,6 +27,7 @@ use datafusion_common::Result as DfResult; /// if one runs right after another. /// /// This rule is expected to be run in the final stage of the optimization process. +#[derive(Debug)] pub struct RemoveDuplicate; impl PhysicalOptimizerRule for RemoveDuplicate { diff --git a/src/query/src/optimizer/scan_hint.rs b/src/query/src/optimizer/scan_hint.rs index e469e7f75b..6be11d63c1 100644 --- a/src/query/src/optimizer/scan_hint.rs +++ b/src/query/src/optimizer/scan_hint.rs @@ -19,7 +19,7 @@ use arrow_schema::SortOptions; use common_recordbatch::OrderOption; use datafusion::datasource::DefaultTableSource; use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeVisitor}; -use datafusion_common::{Column, Result as DataFusionResult}; +use datafusion_common::{Column, Result}; use datafusion_expr::expr::Sort; use datafusion_expr::{utils, Expr, LogicalPlan}; use datafusion_optimizer::{OptimizerConfig, OptimizerRule}; @@ -34,40 +34,36 @@ use crate::dummy_catalog::DummyTableProvider; /// time series row selector hint. /// /// [`ScanRequest`]: store_api::storage::ScanRequest +#[derive(Debug)] pub struct ScanHintRule; impl OptimizerRule for ScanHintRule { - fn try_optimize( - &self, - plan: &LogicalPlan, - _config: &dyn OptimizerConfig, - ) -> DataFusionResult> { - Self::optimize(plan).map(Some) - } - fn name(&self) -> &str { "ScanHintRule" } + + fn rewrite( + &self, + plan: LogicalPlan, + _config: &dyn OptimizerConfig, + ) -> Result> { + Self::optimize(plan) + } } impl ScanHintRule { - fn optimize(plan: &LogicalPlan) -> DataFusionResult { + fn optimize(plan: LogicalPlan) -> Result> { let mut visitor = ScanHintVisitor::default(); let _ = plan.visit(&mut visitor)?; if visitor.need_rewrite() { - plan.clone() - .transform_down(&|plan| Self::set_hints(plan, &visitor)) - .map(|x| x.data) + plan.transform_down(&|plan| Self::set_hints(plan, &visitor)) } else { - Ok(plan.clone()) + Ok(Transformed::no(plan)) } } - fn set_hints( - plan: LogicalPlan, - visitor: &ScanHintVisitor, - ) -> DataFusionResult> { + fn set_hints(plan: LogicalPlan, visitor: &ScanHintVisitor) -> Result> { match &plan { LogicalPlan::TableScan(table_scan) => { let mut transformed = false; @@ -175,16 +171,10 @@ struct ScanHintVisitor { impl TreeNodeVisitor<'_> for ScanHintVisitor { type Node = LogicalPlan; - fn f_down(&mut self, node: &Self::Node) -> DataFusionResult { + fn f_down(&mut self, node: &Self::Node) -> Result { // Get order requirement from sort plan if let LogicalPlan::Sort(sort) = node { - let mut exprs = vec![]; - for expr in &sort.expr { - if let Expr::Sort(sort_expr) = expr { - exprs.push(sort_expr.clone()); - } - } - self.order_expr = Some(exprs); + self.order_expr = Some(sort.expr.clone()); } // Get time series row selector from aggr plan @@ -197,7 +187,7 @@ impl TreeNodeVisitor<'_> for ScanHintVisitor { is_all_last_value = false; break; }; - if func.func_def.name() != "last_value" || func.filter.is_some() || func.distinct { + if func.func.name() != "last_value" || func.filter.is_some() || func.distinct { is_all_last_value = false; break; } @@ -211,10 +201,10 @@ impl TreeNodeVisitor<'_> for ScanHintVisitor { is_all_last_value = false; break; } - } else if let Expr::Sort(sort_expr) = first_order_by { + } else { // only allow `order by xxx [ASC]`, xxx is a bare column reference so `last_value()` is the max // value of the column. - if !sort_expr.asc || !matches!(&*sort_expr.expr, Expr::Column(_)) { + if !first_order_by.asc || !matches!(&first_order_by.expr, Expr::Column(_)) { is_all_last_value = false; break; } @@ -235,13 +225,8 @@ impl TreeNodeVisitor<'_> for ScanHintVisitor { } } // Safety: checked in the above loop - let Expr::Sort(Sort { - expr: order_by_col, .. - }) = order_by_expr.unwrap() - else { - unreachable!() - }; - let Expr::Column(order_by_col) = *order_by_col else { + let order_by_expr = order_by_expr.unwrap(); + let Expr::Column(order_by_col) = order_by_expr.expr else { unreachable!() }; if is_all_last_value { @@ -282,10 +267,10 @@ impl ScanHintVisitor { mod test { use std::sync::Arc; - use datafusion_expr::expr::{AggregateFunction, AggregateFunctionDefinition}; + use datafusion::functions_aggregate::first_last::last_value_udaf; + use datafusion_expr::expr::AggregateFunction; use datafusion_expr::{col, LogicalPlanBuilder}; use datafusion_optimizer::OptimizerContext; - use datafusion_physical_expr::expressions::LastValue; use store_api::storage::RegionId; use super::*; @@ -305,7 +290,8 @@ mod test { .unwrap(); let context = OptimizerContext::default(); - ScanHintRule.try_optimize(&plan, &context).unwrap(); + assert!(ScanHintRule.supports_rewrite()); + ScanHintRule.rewrite(plan, &context).unwrap(); // should read the first (with `.sort(true, false)`) sort option let scan_req = provider.scan_request(); @@ -330,15 +316,15 @@ mod test { .aggregate( vec![col("k0")], vec![Expr::AggregateFunction(AggregateFunction { - func_def: AggregateFunctionDefinition::UDF(Arc::new(LastValue::new().into())), + func: last_value_udaf(), args: vec![col("v0")], distinct: false, filter: None, - order_by: Some(vec![Expr::Sort(Sort { - expr: Box::new(col("ts")), + order_by: Some(vec![Sort { + expr: col("ts"), asc: true, nulls_first: true, - })]), + }]), null_treatment: None, })], ) @@ -347,7 +333,8 @@ mod test { .unwrap(); let context = OptimizerContext::default(); - ScanHintRule.try_optimize(&plan, &context).unwrap(); + assert!(ScanHintRule.supports_rewrite()); + ScanHintRule.rewrite(plan, &context).unwrap(); let scan_req = provider.scan_request(); let _ = scan_req.series_row_selector.unwrap(); diff --git a/src/query/src/optimizer/string_normalization.rs b/src/query/src/optimizer/string_normalization.rs index 1a755f33a2..7dbba9f4f7 100644 --- a/src/query/src/optimizer/string_normalization.rs +++ b/src/query/src/optimizer/string_normalization.rs @@ -20,8 +20,11 @@ use datafusion_common::{Result, ScalarValue}; use datafusion_expr::{Expr, LogicalPlan}; use datafusion_optimizer::analyzer::AnalyzerRule; +use crate::plan::ExtractExpr; + /// StringNormalizationRule normalizes(trims) string values in logical plan. /// Mainly used for timestamp trimming +#[derive(Debug)] pub struct StringNormalizationRule; impl AnalyzerRule for StringNormalizationRule { @@ -30,7 +33,7 @@ impl AnalyzerRule for StringNormalizationRule { let mut converter = StringNormalizationConverter; let inputs = plan.inputs().into_iter().cloned().collect::>(); let expr = plan - .expressions() + .expressions_consider_join() .into_iter() .map(|e| e.rewrite(&mut converter).map(|x| x.data)) .collect::>>()?; @@ -112,7 +115,7 @@ mod tests { let expected = format!("Projection: CAST(Utf8(\"2017-07-23 13:10:11\") AS Timestamp({:#?}, None))\n TableScan: t", time_unit ); - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } } @@ -130,7 +133,7 @@ mod tests { let expected = String::from( "Projection: CAST(Int64(158412331400600000) AS Timestamp(Nanosecond, None))\n TableScan: t" ); - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); let proj_string_to_int = vec![Expr::Cast(Cast::new( Box::new(lit(" 5 ")), @@ -141,7 +144,7 @@ mod tests { .analyze(string_to_int_plan, &ConfigOptions::default()) .unwrap(); let expected = String::from("Projection: CAST(Utf8(\" 5 \") AS Int32)\n TableScan: t"); - assert_eq!(expected, format!("{:?}", result)); + assert_eq!(expected, result.to_string()); } fn create_test_plan_with_project(proj: Vec) -> LogicalPlan { @@ -155,7 +158,7 @@ mod tests { fn create_timestamp_cast_project(unit: TimeUnit, timestamp_str: &str) -> (TimeUnit, Vec) { let proj = vec![Expr::Cast(Cast::new( Box::new(lit(timestamp_str)), - DataType::Timestamp(unit.clone(), None), + DataType::Timestamp(unit, None), ))]; (unit, proj) } diff --git a/src/query/src/optimizer/test_util.rs b/src/query/src/optimizer/test_util.rs index 58e014afff..c57fb9bf4d 100644 --- a/src/query/src/optimizer/test_util.rs +++ b/src/query/src/optimizer/test_util.rs @@ -112,6 +112,7 @@ impl RegionEngine for MetaRegionEngine { } /// Mock a [DummyTableProvider] with a single region. +/// The schema is: `k0: string tag, ts: timestamp, v0: float64 field` pub(crate) fn mock_table_provider(region_id: RegionId) -> DummyTableProvider { let metadata = Arc::new(mock_region_metadata(region_id)); let engine = Arc::new(MetaRegionEngine::with_metadata(metadata.clone())); @@ -119,7 +120,7 @@ pub(crate) fn mock_table_provider(region_id: RegionId) -> DummyTableProvider { } /// Returns a mock region metadata. -/// The schema is: `k0: string, ts: timestamp, v0: float64` +/// The schema is: `k0: string tag, ts: timestamp, v0: float64 field` fn mock_region_metadata(region_id: RegionId) -> RegionMetadata { let mut builder = RegionMetadataBuilder::new(region_id); builder diff --git a/src/query/src/optimizer/type_conversion.rs b/src/query/src/optimizer/type_conversion.rs index c3260c4198..3e6041f1e5 100644 --- a/src/query/src/optimizer/type_conversion.rs +++ b/src/query/src/optimizer/type_conversion.rs @@ -26,6 +26,7 @@ use datatypes::arrow::datatypes::DataType; use session::context::QueryContextRef; use crate::optimizer::ExtensionAnalyzerRule; +use crate::plan::ExtractExpr; use crate::QueryEngineContext; /// TypeConversionRule converts some literal values in logical plan to other types according @@ -88,7 +89,6 @@ impl ExtensionAnalyzerRule for TypeConversionRule { | LogicalPlan::Limit { .. } | LogicalPlan::Union { .. } | LogicalPlan::Join { .. } - | LogicalPlan::CrossJoin { .. } | LogicalPlan::Distinct { .. } | LogicalPlan::Values { .. } | LogicalPlan::Analyze { .. } => { @@ -98,7 +98,7 @@ impl ExtensionAnalyzerRule for TypeConversionRule { }; let inputs = plan.inputs().into_iter().cloned().collect::>(); let expr = plan - .expressions() + .expressions_consider_join() .into_iter() .map(|e| e.rewrite(&mut converter).map(|x| x.data)) .collect::>>()?; @@ -110,7 +110,6 @@ impl ExtensionAnalyzerRule for TypeConversionRule { | LogicalPlan::Explain { .. } | LogicalPlan::SubqueryAlias { .. } | LogicalPlan::EmptyRelation(_) - | LogicalPlan::Prepare(_) | LogicalPlan::Dml(_) | LogicalPlan::DescribeTable(_) | LogicalPlan::Unnest(_) @@ -307,11 +306,9 @@ mod tests { use std::collections::HashMap; use std::sync::Arc; - use datafusion::logical_expr::expr::AggregateFunction as AggrExpr; use datafusion_common::arrow::datatypes::Field; use datafusion_common::{Column, DFSchema}; - use datafusion_expr::expr::AggregateFunctionDefinition; - use datafusion_expr::{AggregateFunction, LogicalPlanBuilder}; + use datafusion_expr::LogicalPlanBuilder; use datafusion_sql::TableReference; use session::context::QueryContext; @@ -476,14 +473,16 @@ mod tests { .unwrap() .aggregate( Vec::::new(), - vec![Expr::AggregateFunction(AggrExpr { - func_def: AggregateFunctionDefinition::BuiltIn(AggregateFunction::Count), - args: vec![Expr::Column(Column::from_name("column1"))], - distinct: false, - filter: None, - order_by: None, - null_treatment: None, - })], + vec![Expr::AggregateFunction( + datafusion_expr::expr::AggregateFunction::new_udf( + datafusion::functions_aggregate::count::count_udaf(), + vec![Expr::Column(Column::from_name("column1"))], + false, + None, + None, + None, + ), + )], ) .unwrap() .build() @@ -494,7 +493,7 @@ mod tests { .analyze(plan, &context, &ConfigOptions::default()) .unwrap(); let expected = String::from( - "Aggregate: groupBy=[[]], aggr=[[COUNT(column1)]]\ + "Aggregate: groupBy=[[]], aggr=[[count(column1)]]\ \n Filter: TimestampSecond(-28800, None) <= column3\ \n Filter: column3 > TimestampSecond(-28800, None)\ \n Values: (Int64(1), Float64(1), TimestampMillisecond(1, None))", diff --git a/src/query/src/optimizer/windowed_sort.rs b/src/query/src/optimizer/windowed_sort.rs index 638c3d9d39..49cde9783b 100644 --- a/src/query/src/optimizer/windowed_sort.rs +++ b/src/query/src/optimizer/windowed_sort.rs @@ -24,6 +24,7 @@ use datafusion::physical_plan::ExecutionPlan; use datafusion_common::tree_node::{Transformed, TreeNode}; use datafusion_common::Result as DataFusionResult; use datafusion_physical_expr::expressions::Column as PhysicalColumn; +use datafusion_physical_expr::LexOrdering; use store_api::region_engine::PartitionRange; use table::table::scan::RegionScanExec; @@ -39,6 +40,7 @@ use crate::window_sort::WindowedSortExec; /// /// [`ScanHint`]: crate::optimizer::scan_hint::ScanHintRule /// [`ParallelizeScan`]: crate::optimizer::parallelize_scan::ParallelizeScan +#[derive(Debug)] pub struct WindowedSortPhysicalRule; impl PhysicalOptimizerRule for WindowedSortPhysicalRule { @@ -116,7 +118,7 @@ impl WindowedSortPhysicalRule { if !preserve_partitioning { let order_preserving_merge = SortPreservingMergeExec::new( - sort_exec.expr().to_vec(), + LexOrdering::new(sort_exec.expr().to_vec()), Arc::new(windowed_sort_exec), ); return Ok(Transformed { diff --git a/src/query/src/parser.rs b/src/query/src/parser.rs index b99bb6953e..7867cc38fc 100644 --- a/src/query/src/parser.rs +++ b/src/query/src/parser.rs @@ -281,35 +281,10 @@ mod test { fn parse_sql_simple() { let sql = "select * from t1"; let stmt = QueryLanguageParser::parse_sql(sql, &QueryContext::arc()).unwrap(); - let expected = String::from("Sql(Query(Query { \ - inner: Query { \ - with: None, body: Select(Select { \ - distinct: None, \ - top: None, \ - projection: \ - [Wildcard(WildcardAdditionalOptions { opt_exclude: None, opt_except: None, opt_rename: None, opt_replace: None })], \ - into: None, \ - from: [TableWithJoins { relation: Table { name: ObjectName([Ident { value: \"t1\", quote_style: None }]\ - ), \ - alias: None, \ - args: None, \ - with_hints: [], \ - version: None, \ - partitions: [] \ - }, \ - joins: [] }], \ - lateral_views: [], \ - selection: None, \ - group_by: Expressions([]), \ - cluster_by: [], \ - distribute_by: [], \ - sort_by: [], \ - having: None, \ - named_window: [], \ - qualify: None, \ - value_table_mode: None \ - }), order_by: [], limit: None, limit_by: [], offset: None, fetch: None, locks: [], for_clause: None } }))"); - assert_eq!(format!("{stmt:?}"), expected); + let QueryStatement::Sql(sql_stmt) = stmt else { + panic!("Expected SQL statement, got {:?}", stmt); + }; + assert_eq!("SELECT * FROM t1", sql_stmt.to_string()); } #[test] diff --git a/src/query/src/part_sort.rs b/src/query/src/part_sort.rs index 9339c16b08..1b5fbfe201 100644 --- a/src/query/src/part_sort.rs +++ b/src/query/src/part_sort.rs @@ -24,19 +24,18 @@ use std::sync::Arc; use std::task::{Context, Poll}; use arrow::array::ArrayRef; -use arrow::compute::{concat, take_record_batch}; +use arrow::compute::{concat, concat_batches, take_record_batch}; use arrow_schema::SchemaRef; use common_recordbatch::{DfRecordBatch, DfSendableRecordBatchStream}; use datafusion::common::arrow::compute::sort_to_indices; use datafusion::execution::memory_pool::{MemoryConsumer, MemoryReservation}; use datafusion::execution::{RecordBatchStream, TaskContext}; -use datafusion::physical_plan::coalesce_batches::concat_batches; use datafusion::physical_plan::metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricsSet}; use datafusion::physical_plan::{ DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, PlanProperties, TopK, }; use datafusion_common::{internal_err, DataFusionError}; -use datafusion_physical_expr::PhysicalSortExpr; +use datafusion_physical_expr::{LexOrdering, PhysicalSortExpr}; use futures::{Stream, StreamExt}; use itertools::Itertools; use snafu::location; @@ -130,6 +129,10 @@ impl DisplayAs for PartSortExec { } impl ExecutionPlan for PartSortExec { + fn name(&self) -> &str { + "PartSortExec" + } + fn as_any(&self) -> &dyn Any { self } @@ -237,12 +240,11 @@ impl PartSortStream { TopK::try_new( partition, sort.schema().clone(), - vec![sort.expression.clone()], + LexOrdering::new(vec![sort.expression.clone()]), limit, context.session_config().batch_size(), context.runtime_env(), &sort.metrics, - partition, )?, 0, ) @@ -459,12 +461,7 @@ impl PartSortStream { let total_mem: usize = buffer.iter().map(|r| r.get_array_memory_size()).sum(); self.reservation.try_grow(total_mem * 2)?; - let full_input = concat_batches( - &self.schema, - &buffer, - buffer.iter().map(|r| r.num_rows()).sum(), - ) - .map_err(|e| { + let full_input = concat_batches(&self.schema, &buffer).map_err(|e| { DataFusionError::ArrowError( e, Some(format!( @@ -496,12 +493,11 @@ impl PartSortStream { let new_top_buffer = TopK::try_new( self.partition, self.schema().clone(), - vec![self.expression.clone()], + LexOrdering::new(vec![self.expression.clone()]), self.limit.unwrap(), self.context.session_config().batch_size(), self.context.runtime_env(), &self.root_metrics, - self.partition, )?; let PartSortBuffer::Top(top_k, _) = std::mem::replace(&mut self.buffer, PartSortBuffer::Top(new_top_buffer, 0)) @@ -512,13 +508,11 @@ impl PartSortStream { let mut result_stream = top_k.emit()?; let mut placeholder_ctx = std::task::Context::from_waker(futures::task::noop_waker_ref()); let mut results = vec![]; - let mut row_count = 0; // according to the current implementation of `TopK`, the result stream will always be ready loop { match result_stream.poll_next_unpin(&mut placeholder_ctx) { Poll::Ready(Some(batch)) => { let batch = batch?; - row_count += batch.num_rows(); results.push(batch); } Poll::Pending => { @@ -531,7 +525,7 @@ impl PartSortStream { } } - let concat_batch = concat_batches(&self.schema, &results, row_count).map_err(|e| { + let concat_batch = concat_batches(&self.schema, &results).map_err(|e| { DataFusionError::ArrowError( e, Some(format!( @@ -718,7 +712,7 @@ mod test { let schema = Schema::new(vec![Field::new( "ts", - DataType::Timestamp(unit.clone(), None), + DataType::Timestamp(unit, None), false, )]); let schema = Arc::new(schema); @@ -739,8 +733,8 @@ mod test { .unwrap_or_else(|| rng.i64(-100000000..100000000)); bound_val = Some(end); let start = end - rng.i64(1..range_size_bound); - let start = Timestamp::new(start, unit.clone().into()); - let end = Timestamp::new(end, unit.clone().into()); + let start = Timestamp::new(start, unit.into()); + let end = Timestamp::new(end, unit.into()); (start, end) } else { let start = bound_val @@ -748,8 +742,8 @@ mod test { .unwrap_or_else(|| rng.i64(..)); bound_val = Some(start); let end = start + rng.i64(1..range_size_bound); - let start = Timestamp::new(start, unit.clone().into()); - let end = Timestamp::new(end, unit.clone().into()); + let start = Timestamp::new(start, unit.into()); + let end = Timestamp::new(end, unit.into()); (start, end) }; assert!(start < end); @@ -769,7 +763,7 @@ mod test { // mito always sort on ASC order data_gen.sort(); per_part_sort_data.extend(data_gen.clone()); - let arr = new_ts_array(unit.clone(), data_gen.clone()); + let arr = new_ts_array(unit, data_gen.clone()); let batch = DfRecordBatch::try_new(schema.clone(), vec![arr]).unwrap(); batches.push(batch); } @@ -816,8 +810,7 @@ mod test { let expected_output = output_data .into_iter() .map(|a| { - DfRecordBatch::try_new(schema.clone(), vec![new_ts_array(unit.clone(), a)]) - .unwrap() + DfRecordBatch::try_new(schema.clone(), vec![new_ts_array(unit, a)]).unwrap() }) .map(|rb| { // trim expected output with limit @@ -954,7 +947,7 @@ mod test { { let schema = Schema::new(vec![Field::new( "ts", - DataType::Timestamp(unit.clone(), None), + DataType::Timestamp(unit, None), false, )]); let schema = Arc::new(schema); @@ -967,8 +960,8 @@ mod test { .into_iter() .map(|(range, data)| { let part = PartitionRange { - start: Timestamp::new(range.0, unit.clone().into()), - end: Timestamp::new(range.1, unit.clone().into()), + start: Timestamp::new(range.0, unit.into()), + end: Timestamp::new(range.1, unit.into()), num_rows: data.iter().map(|b| b.len()).sum(), identifier, }; @@ -976,7 +969,7 @@ mod test { let batches = data .into_iter() .map(|b| { - let arr = new_ts_array(unit.clone(), b); + let arr = new_ts_array(unit, b); DfRecordBatch::try_new(schema.clone(), vec![arr]).unwrap() }) .collect_vec(); @@ -987,8 +980,7 @@ mod test { let expected_output = expected_output .into_iter() .map(|a| { - DfRecordBatch::try_new(schema.clone(), vec![new_ts_array(unit.clone(), a)]) - .unwrap() + DfRecordBatch::try_new(schema.clone(), vec![new_ts_array(unit, a)]).unwrap() }) .collect_vec(); diff --git a/src/query/src/plan.rs b/src/query/src/plan.rs index 710e92129d..e94c073c70 100644 --- a/src/query/src/plan.rs +++ b/src/query/src/plan.rs @@ -17,7 +17,7 @@ use std::collections::HashSet; use datafusion::datasource::DefaultTableSource; use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRewriter}; use datafusion_common::TableReference; -use datafusion_expr::LogicalPlan as DfLogicalPlan; +use datafusion_expr::{BinaryExpr, Expr, Join, LogicalPlan, Operator}; use session::context::QueryContextRef; use snafu::ResultExt; pub use table::metadata::TableType; @@ -32,7 +32,7 @@ struct TableNamesExtractAndRewriter { } impl TreeNodeRewriter for TableNamesExtractAndRewriter { - type Node = DfLogicalPlan; + type Node = LogicalPlan; /// descend fn f_down<'a>( @@ -40,7 +40,7 @@ impl TreeNodeRewriter for TableNamesExtractAndRewriter { node: Self::Node, ) -> datafusion::error::Result> { match node { - DfLogicalPlan::TableScan(mut scan) => { + LogicalPlan::TableScan(mut scan) => { if let Some(source) = scan.source.as_any().downcast_ref::() { if let Some(provider) = source .table_provider @@ -96,7 +96,7 @@ impl TreeNodeRewriter for TableNamesExtractAndRewriter { }; } } - Ok(Transformed::yes(DfLogicalPlan::TableScan(scan))) + Ok(Transformed::yes(LogicalPlan::TableScan(scan))) } node => Ok(Transformed::no(node)), } @@ -115,14 +115,45 @@ impl TableNamesExtractAndRewriter { /// Extracts and rewrites the table names in the plan in the fully qualified style, /// return the table names and new plan. pub fn extract_and_rewrite_full_table_names( - plan: DfLogicalPlan, + plan: LogicalPlan, query_ctx: QueryContextRef, -) -> Result<(HashSet, DfLogicalPlan)> { +) -> Result<(HashSet, LogicalPlan)> { let mut extractor = TableNamesExtractAndRewriter::new(query_ctx); let plan = plan.rewrite(&mut extractor).context(DataFusionSnafu)?; Ok((extractor.table_names, plan.data)) } +/// A trait to extract expressions from a logical plan. +pub trait ExtractExpr { + /// Gets expressions from a logical plan. + /// It handles [Join] specially so [LogicalPlan::with_new_exprs()] can use the expressions + /// this method returns. + fn expressions_consider_join(&self) -> Vec; +} + +impl ExtractExpr for LogicalPlan { + fn expressions_consider_join(&self) -> Vec { + match self { + LogicalPlan::Join(Join { on, filter, .. }) => { + // The first part of expr is equi-exprs, + // and the struct of each equi-expr is like `left-expr = right-expr`. + // We only normalize the filter_expr (non equality predicate from ON clause). + on.iter() + .map(|(left, right)| { + Expr::BinaryExpr(BinaryExpr { + left: Box::new(left.clone()), + op: Operator::Eq, + right: Box::new(right.clone()), + }) + }) + .chain(filter.clone()) + .collect() + } + _ => self.expressions(), + } + } +} + #[cfg(test)] pub(crate) mod tests { @@ -136,7 +167,7 @@ pub(crate) mod tests { use super::*; - pub(crate) fn mock_plan() -> LogicalPlan { + fn mock_plan() -> LogicalPlan { let schema = Schema::new(vec![ Field::new("id", DataType::Int32, true), Field::new("name", DataType::Utf8, true), @@ -174,7 +205,7 @@ pub(crate) mod tests { assert_eq!( "Filter: devices.id > Int32(500)\n TableScan: greptime.test.devices", - format!("{:?}", plan) + plan.to_string() ); } } diff --git a/src/query/src/planner.rs b/src/query/src/planner.rs index 20377c67c0..75d70e5901 100644 --- a/src/query/src/planner.rs +++ b/src/query/src/planner.rs @@ -98,6 +98,10 @@ impl DfLogicalPlanner { let parser_options = ParserOptions { enable_ident_normalization: config_options.sql_parser.enable_ident_normalization, parse_float_as_decimal: config_options.sql_parser.parse_float_as_decimal, + support_varchar_with_length: config_options.sql_parser.support_varchar_with_length, + enable_options_value_normalization: config_options + .sql_parser + .enable_options_value_normalization, }; let sql_to_rel = SqlToRel::new_with_options(&context_provider, parser_options); @@ -140,6 +144,10 @@ impl DfLogicalPlanner { let parser_options = ParserOptions { enable_ident_normalization: normalize_ident, parse_float_as_decimal: config_options.sql_parser.parse_float_as_decimal, + support_varchar_with_length: config_options.sql_parser.support_varchar_with_length, + enable_options_value_normalization: config_options + .sql_parser + .enable_options_value_normalization, }; let sql_to_rel = SqlToRel::new_with_options(&context_provider, parser_options); diff --git a/src/query/src/promql/planner.rs b/src/query/src/promql/planner.rs index 48a88a2cb5..69b4aef6d5 100644 --- a/src/query/src/promql/planner.rs +++ b/src/query/src/promql/planner.rs @@ -16,26 +16,32 @@ use std::collections::{BTreeSet, HashSet, VecDeque}; use std::sync::Arc; use std::time::UNIX_EPOCH; +use arrow::datatypes::IntervalDayTime; use async_recursion::async_recursion; use catalog::table_source::DfTableSourceProvider; use common_query::prelude::GREPTIME_VALUE; -use datafusion::common::{DFSchemaRef, Result as DfResult}; +use datafusion::common::DFSchemaRef; use datafusion::datasource::DefaultTableSource; use datafusion::execution::context::SessionState; -use datafusion::functions_aggregate::sum; -use datafusion::logical_expr::expr::{ - AggregateFunction, AggregateFunctionDefinition, Alias, ScalarFunction, -}; +use datafusion::functions_aggregate::average::avg_udaf; +use datafusion::functions_aggregate::count::count_udaf; +use datafusion::functions_aggregate::grouping::grouping_udaf; +use datafusion::functions_aggregate::min_max::{max_udaf, min_udaf}; +use datafusion::functions_aggregate::stddev::stddev_pop_udaf; +use datafusion::functions_aggregate::sum::sum_udaf; +use datafusion::functions_aggregate::variance::var_pop_udaf; +use datafusion::logical_expr::expr::{AggregateFunction, Alias, ScalarFunction}; use datafusion::logical_expr::expr_rewriter::normalize_cols; use datafusion::logical_expr::{ - AggregateFunction as AggregateFunctionEnum, BinaryExpr, Cast, Extension, LogicalPlan, - LogicalPlanBuilder, Operator, ScalarUDF as ScalarUdfDef, + BinaryExpr, Cast, Extension, LogicalPlan, LogicalPlanBuilder, Operator, + ScalarUDF as ScalarUdfDef, }; use datafusion::prelude as df_prelude; use datafusion::prelude::{Column, Expr as DfExpr, JoinType}; use datafusion::scalar::ScalarValue; use datafusion::sql::TableReference; use datafusion_expr::utils::conjunction; +use datafusion_expr::SortExpr; use datatypes::arrow::datatypes::{DataType as ArrowDataType, TimeUnit as ArrowTimeUnit}; use datatypes::data_type::ConcreteDataType; use itertools::Itertools; @@ -1322,8 +1328,9 @@ impl PromPlanner { let month_lit_expr = DfExpr::Literal(ScalarValue::Utf8(Some("month".to_string()))); let interval_1month_lit_expr = DfExpr::Literal(ScalarValue::IntervalYearMonth(Some(1))); - let interval_1day_lit_expr = - DfExpr::Literal(ScalarValue::IntervalDayTime(Some(1 << 32))); + let interval_1day_lit_expr = DfExpr::Literal(ScalarValue::IntervalDayTime(Some( + IntervalDayTime::new(1, 0), + ))); let the_1month_minus_1day_expr = DfExpr::BinaryExpr(BinaryExpr { left: Box::new(interval_1month_lit_expr), op: Operator::Minus, @@ -1471,7 +1478,7 @@ impl PromPlanner { exprs = exprs .into_iter() .map(|expr| { - let display_name = expr.display_name()?; + let display_name = expr.schema_name().to_string(); new_field_columns.push(display_name.clone()); Ok(expr.alias(display_name)) }) @@ -1635,7 +1642,7 @@ impl PromPlanner { Ok(result) } - fn create_tag_and_time_index_column_sort_exprs(&self) -> Result> { + fn create_tag_and_time_index_column_sort_exprs(&self) -> Result> { let mut result = self .ctx .tag_columns @@ -1670,18 +1677,14 @@ impl PromPlanner { input_plan: &LogicalPlan, ) -> Result> { let aggr = match op.id() { - token::T_SUM => AggregateFunctionDefinition::UDF(sum::sum_udaf()), - token::T_AVG => AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::Avg), - token::T_COUNT => AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::Count), - token::T_MIN => AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::Min), - token::T_MAX => AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::Max), - token::T_GROUP => AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::Grouping), - token::T_STDDEV => { - AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::StddevPop) - } - token::T_STDVAR => { - AggregateFunctionDefinition::BuiltIn(AggregateFunctionEnum::VariancePop) - } + token::T_SUM => sum_udaf(), + token::T_AVG => avg_udaf(), + token::T_COUNT => count_udaf(), + token::T_MIN => min_udaf(), + token::T_MAX => max_udaf(), + token::T_GROUP => grouping_udaf(), + token::T_STDDEV => stddev_pop_udaf(), + token::T_STDVAR => var_pop_udaf(), token::T_TOPK | token::T_BOTTOMK | token::T_COUNT_VALUES | token::T_QUANTILE => { UnsupportedExprSnafu { name: format!("{op:?}"), @@ -1698,7 +1701,7 @@ impl PromPlanner { .iter() .map(|col| { DfExpr::AggregateFunction(AggregateFunction { - func_def: aggr.clone(), + func: aggr.clone(), args: vec![DfExpr::Column(Column::from_name(col))], distinct: false, filter: None, @@ -1713,7 +1716,7 @@ impl PromPlanner { let normalized_exprs = normalize_cols(exprs.iter().cloned(), input_plan).context(DataFusionPlanningSnafu)?; for expr in normalized_exprs { - new_field_columns.push(expr.display_name().context(DataFusionPlanningSnafu)?); + new_field_columns.push(expr.schema_name().to_string()); } self.ctx.field_columns = new_field_columns; @@ -2403,9 +2406,8 @@ impl PromPlanner { // alias the computation exprs to remove qualifier self.ctx.field_columns = result_field_columns .iter() - .map(|expr| expr.display_name()) - .collect::>>() - .context(DataFusionPlanningSnafu)?; + .map(|expr| expr.schema_name().to_string()) + .collect(); let field_columns_iter = result_field_columns .into_iter() .zip(self.ctx.field_columns.iter()) @@ -2497,10 +2499,9 @@ mod test { use catalog::RegisterTableRequest; use common_catalog::consts::{DEFAULT_CATALOG_NAME, DEFAULT_SCHEMA_NAME}; use common_query::test_util::DummyDecoder; - use datafusion::execution::runtime_env::RuntimeEnv; + use datafusion::execution::SessionStateBuilder; use datatypes::prelude::ConcreteDataType; use datatypes::schema::{ColumnSchema, Schema}; - use df_prelude::SessionConfig; use promql_parser::label::Labels; use promql_parser::parser; use session::context::QueryContext; @@ -2510,7 +2511,7 @@ mod test { use super::*; fn build_session_state() -> SessionState { - SessionState::new_with_config_rt(SessionConfig::new(), Arc::new(RuntimeEnv::default())) + SessionStateBuilder::new().with_default_features().build() } async fn build_test_table_provider( @@ -2870,28 +2871,28 @@ mod test { #[tokio::test] async fn aggregate_sum() { - do_aggregate_expr_plan("sum", "SUM").await; + do_aggregate_expr_plan("sum", "sum").await; } #[tokio::test] async fn aggregate_avg() { - do_aggregate_expr_plan("avg", "AVG").await; + do_aggregate_expr_plan("avg", "avg").await; } #[tokio::test] #[should_panic] // output type doesn't match async fn aggregate_count() { - do_aggregate_expr_plan("count", "COUNT").await; + do_aggregate_expr_plan("count", "count").await; } #[tokio::test] async fn aggregate_min() { - do_aggregate_expr_plan("min", "MIN").await; + do_aggregate_expr_plan("min", "min").await; } #[tokio::test] async fn aggregate_max() { - do_aggregate_expr_plan("max", "MAX").await; + do_aggregate_expr_plan("max", "max").await; } #[tokio::test] @@ -2902,12 +2903,12 @@ mod test { #[tokio::test] async fn aggregate_stddev() { - do_aggregate_expr_plan("stddev", "STDDEV_POP").await; + do_aggregate_expr_plan("stddev", "stddev_pop").await; } #[tokio::test] async fn aggregate_stdvar() { - do_aggregate_expr_plan("stdvar", "VAR_POP").await; + do_aggregate_expr_plan("stdvar", "var_pop").await; } #[tokio::test] diff --git a/src/query/src/query_engine/default_serializer.rs b/src/query/src/query_engine/default_serializer.rs index 5c7e01db85..cd04fb5d69 100644 --- a/src/query/src/query_engine/default_serializer.rs +++ b/src/query/src/query_engine/default_serializer.rs @@ -17,15 +17,16 @@ use std::sync::Arc; use common_error::ext::BoxedError; use common_function::function_registry::FUNCTION_REGISTRY; use common_function::scalars::udf::create_udf; +use common_query::error::RegisterUdfSnafu; use common_query::logical_plan::SubstraitPlanDecoder; use datafusion::catalog::CatalogProviderList; use datafusion::common::DataFusionError; use datafusion::error::Result; use datafusion::execution::context::SessionState; use datafusion::execution::registry::SerializerRegistry; -use datafusion::execution::FunctionRegistry; +use datafusion::execution::{FunctionRegistry, SessionStateBuilder}; use datafusion::logical_expr::LogicalPlan; -use datafusion_expr::UserDefinedLogicalNode; +use datafusion_expr::{ScalarUDF, UserDefinedLogicalNode}; use greptime_proto::substrait_extension::MergeScan as PbMergeScan; use prost::Message; use session::context::QueryContextRef; @@ -34,9 +35,9 @@ use substrait::extension_serializer::ExtensionSerializer; use substrait::{DFLogicalSubstraitConvertor, SubstraitPlan}; use crate::dist_plan::MergeScanLogicalPlan; -use crate::error::DataFusionSnafu; /// Extended [`substrait::extension_serializer::ExtensionSerializer`] but supports [`MergeScanLogicalPlan`] serialization. +#[derive(Debug)] pub struct DefaultSerializer; impl SerializerRegistry for DefaultSerializer { @@ -84,21 +85,18 @@ impl SerializerRegistry for DefaultSerializer { /// The datafusion `[LogicalPlan]` decoder. pub struct DefaultPlanDecoder { session_state: SessionState, + query_ctx: QueryContextRef, } impl DefaultPlanDecoder { pub fn new( - mut session_state: SessionState, + session_state: SessionState, query_ctx: &QueryContextRef, ) -> crate::error::Result { - // Substrait decoder will look up the UDFs in SessionState, so we need to register them - // Note: the query context must be passed to set the timezone - for func in FUNCTION_REGISTRY.functions() { - let udf = Arc::new(create_udf(func, query_ctx.clone(), Default::default()).into()); - session_state.register_udf(udf).context(DataFusionSnafu)?; - } - - Ok(Self { session_state }) + Ok(Self { + session_state, + query_ctx: query_ctx.clone(), + }) } } @@ -111,8 +109,25 @@ impl SubstraitPlanDecoder for DefaultPlanDecoder { optimize: bool, ) -> common_query::error::Result { // The session_state already has the `DefaultSerialzier` as `SerializerRegistry`. + let mut session_state = SessionStateBuilder::new_from_existing(self.session_state.clone()) + .with_catalog_list(catalog_list) + .build(); + // Substrait decoder will look up the UDFs in SessionState, so we need to register them + // Note: the query context must be passed to set the timezone + // We MUST register the UDFs after we build the session state, otherwise the UDFs will be lost + // if they have the same name as the default UDFs or their alias. + // e.g. The default UDF `to_char()` has an alias `date_format()`, if we register a UDF with the name `date_format()` + // before we build the session state, the UDF will be lost. + for func in FUNCTION_REGISTRY.functions() { + let udf: Arc = Arc::new( + create_udf(func.clone(), self.query_ctx.clone(), Default::default()).into(), + ); + session_state + .register_udf(udf) + .context(RegisterUdfSnafu { name: func.name() })?; + } let logical_plan = DFLogicalSubstraitConvertor - .decode(message, catalog_list.clone(), self.session_state.clone()) + .decode(message, session_state) .await .map_err(BoxedError::new) .context(common_query::error::DecodePlanSnafu)?; @@ -129,14 +144,29 @@ impl SubstraitPlanDecoder for DefaultPlanDecoder { #[cfg(test)] mod tests { + use datafusion::catalog::TableProvider; + use datafusion_expr::{col, lit, LogicalPlanBuilder, LogicalTableSource}; + use datatypes::arrow::datatypes::SchemaRef; use session::context::QueryContext; use super::*; use crate::dummy_catalog::DummyCatalogList; use crate::optimizer::test_util::mock_table_provider; - use crate::plan::tests::mock_plan; use crate::QueryEngineFactory; + fn mock_plan(schema: SchemaRef) -> LogicalPlan { + let table_source = LogicalTableSource::new(schema); + let projection = None; + let builder = + LogicalPlanBuilder::scan("devices", Arc::new(table_source), projection).unwrap(); + + builder + .filter(col("k0").eq(lit("hello"))) + .unwrap() + .build() + .unwrap() + } + #[tokio::test] async fn test_serializer_decode_plan() { let catalog_list = catalog::memory::new_memory_catalog_manager().unwrap(); @@ -144,7 +174,8 @@ mod tests { let engine = factory.query_engine(); - let plan = mock_plan(); + let table_provider = Arc::new(mock_table_provider(1.into())); + let plan = mock_plan(table_provider.schema().clone()); let bytes = DFLogicalSubstraitConvertor .encode(&plan, DefaultSerializer) @@ -154,7 +185,6 @@ mod tests { .engine_context(QueryContext::arc()) .new_plan_decoder() .unwrap(); - let table_provider = Arc::new(mock_table_provider(1.into())); let catalog_list = Arc::new(DummyCatalogList::with_table_provider(table_provider)); let decode_plan = plan_decoder @@ -163,9 +193,9 @@ mod tests { .unwrap(); assert_eq!( - "Filter: devices.k0 > Int32(500) - TableScan: devices projection=[k0, ts, v0]", - format!("{:?}", decode_plan), + "Filter: devices.k0 = Utf8(\"hello\") + TableScan: devices", + decode_plan.to_string(), ); } } diff --git a/src/query/src/query_engine/state.rs b/src/query/src/query_engine/state.rs index 74db773031..87674866d2 100644 --- a/src/query/src/query_engine/state.rs +++ b/src/query/src/query_engine/state.rs @@ -31,8 +31,9 @@ use datafusion::dataframe::DataFrame; use datafusion::error::Result as DfResult; use datafusion::execution::context::{QueryPlanner, SessionConfig, SessionContext, SessionState}; use datafusion::execution::runtime_env::RuntimeEnv; +use datafusion::execution::SessionStateBuilder; use datafusion::physical_optimizer::optimizer::PhysicalOptimizer; -use datafusion::physical_optimizer::pipeline_checker::PipelineChecker; +use datafusion::physical_optimizer::sanity_checker::SanityCheckPlan; use datafusion::physical_optimizer::PhysicalOptimizerRule; use datafusion::physical_plan::ExecutionPlan; use datafusion::physical_planner::{DefaultPhysicalPlanner, ExtensionPlanner, PhysicalPlanner}; @@ -60,9 +61,6 @@ use crate::region_query::RegionQueryHandlerRef; use crate::QueryEngineContext; /// Query engine global state -// TODO(yingwen): This QueryEngineState still relies on datafusion, maybe we can define a trait for it, -// which allows different implementation use different engine state. The state can also be an associated -// type in QueryEngine trait. #[derive(Clone)] pub struct QueryEngineState { df_context: SessionContext, @@ -129,14 +127,17 @@ impl QueryEngineState { .push(Arc::new(WindowedSortPhysicalRule)); // Add rule to remove duplicate nodes generated by other rules. Run this in the last. physical_optimizer.rules.push(Arc::new(RemoveDuplicate)); - // Place PipelineChecker at the end of the list to ensure that it runs after all other rules. + // Place SanityCheckPlan at the end of the list to ensure that it runs after all other rules. Self::remove_physical_optimizer_rule( &mut physical_optimizer.rules, - PipelineChecker {}.name(), + SanityCheckPlan {}.name(), ); - physical_optimizer.rules.push(Arc::new(PipelineChecker {})); + physical_optimizer.rules.push(Arc::new(SanityCheckPlan {})); - let session_state = SessionState::new_with_config_rt(session_config, runtime_env) + let session_state = SessionStateBuilder::new() + .with_config(session_config) + .with_runtime_env(runtime_env) + .with_default_features() .with_analyzer_rules(analyzer.rules) .with_serializer_registry(Arc::new(DefaultSerializer)) .with_query_planner(Arc::new(DfQueryPlanner::new( @@ -144,7 +145,8 @@ impl QueryEngineState { region_query_handler, ))) .with_optimizer_rules(optimizer.rules) - .with_physical_optimizer_rules(physical_optimizer.rules); + .with_physical_optimizer_rules(physical_optimizer.rules) + .build(); let df_context = SessionContext::new_with_state(session_state); @@ -216,6 +218,11 @@ impl QueryEngineState { .cloned() } + /// Retrieve udf function names. + pub fn udf_names(&self) -> Vec { + self.udf_functions.read().unwrap().keys().cloned().collect() + } + /// Retrieve the aggregate function by name pub fn aggregate_function(&self, function_name: &str) -> Option { self.aggregate_functions @@ -225,6 +232,16 @@ impl QueryEngineState { .cloned() } + /// Retrieve aggregate function names. + pub fn udaf_names(&self) -> Vec { + self.aggregate_functions + .read() + .unwrap() + .keys() + .cloned() + .collect() + } + /// Register a [`ScalarUdf`]. pub fn register_udf(&self, udf: ScalarUdf) { self.df_context.register_udf(udf.into()); @@ -290,6 +307,12 @@ struct DfQueryPlanner { physical_planner: DefaultPhysicalPlanner, } +impl fmt::Debug for DfQueryPlanner { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + f.debug_struct("DfQueryPlanner").finish() + } +} + #[async_trait] impl QueryPlanner for DfQueryPlanner { async fn create_physical_plan( diff --git a/src/query/src/range_select/plan.rs b/src/query/src/range_select/plan.rs index 1d2d586860..859a6260d9 100644 --- a/src/query/src/range_select/plan.rs +++ b/src/query/src/range_select/plan.rs @@ -12,7 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::any::Any; use std::cmp::Ordering; use std::collections::btree_map::Entry; use std::collections::{BTreeMap, HashMap}; @@ -23,7 +22,7 @@ use std::task::{Context, Poll}; use std::time::Duration; use ahash::RandomState; -use arrow::compute::{self, cast_with_options, CastOptions, SortColumn}; +use arrow::compute::{self, cast_with_options, take_arrays, CastOptions}; use arrow_schema::{DataType, Field, Schema, SchemaRef, SortOptions, TimeUnit}; use common_recordbatch::DfSendableRecordBatchStream; use datafusion::common::{Result as DataFusionResult, Statistics}; @@ -31,25 +30,20 @@ use datafusion::error::Result as DfResult; use datafusion::execution::context::SessionState; use datafusion::execution::TaskContext; use datafusion::physical_plan::metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricsSet}; -use datafusion::physical_plan::udaf::create_aggregate_expr as create_aggr_udf_expr; use datafusion::physical_plan::{ DisplayAs, DisplayFormatType, ExecutionMode, ExecutionPlan, PlanProperties, RecordBatchStream, SendableRecordBatchStream, }; use datafusion::physical_planner::create_physical_sort_expr; use datafusion_common::hash_utils::create_hashes; -use datafusion_common::utils::{get_arrayref_at_indices, get_row_at_idx}; use datafusion_common::{DFSchema, DFSchemaRef, DataFusionError, ScalarValue}; -use datafusion_expr::expr::AggregateFunctionDefinition; use datafusion_expr::utils::{exprlist_to_fields, COUNT_STAR_EXPANSION}; use datafusion_expr::{ - lit, Accumulator, AggregateFunction, Expr, ExprSchemable, LogicalPlan, - UserDefinedLogicalNodeCore, + lit, Accumulator, Expr, ExprSchemable, LogicalPlan, UserDefinedLogicalNodeCore, }; -use datafusion_physical_expr::aggregate::utils::down_cast_any_ref; -use datafusion_physical_expr::expressions::create_aggregate_expr as create_aggr_expr; +use datafusion_physical_expr::aggregate::{AggregateExprBuilder, AggregateFunctionExpr}; use datafusion_physical_expr::{ - create_physical_expr, AggregateExpr, Distribution, EquivalenceProperties, Partitioning, + create_physical_expr, Distribution, EquivalenceProperties, LexOrdering, Partitioning, PhysicalExpr, PhysicalSortExpr, }; use datatypes::arrow::array::{ @@ -66,140 +60,6 @@ use crate::error::{DataFusionSnafu, RangeQuerySnafu, Result}; type Millisecond = ::Native; -/// Implementation of `first_value`/`last_value` -/// aggregate function adapted to range query -#[derive(Debug)] -struct RangeFirstListValue { - /// calculate expr - expr: Arc, - order_bys: Vec, -} - -impl RangeFirstListValue { - pub fn new_aggregate_expr( - expr: Arc, - order_bys: Vec, - ) -> Arc { - Arc::new(Self { expr, order_bys }) - } -} - -impl PartialEq for RangeFirstListValue { - fn eq(&self, other: &dyn Any) -> bool { - down_cast_any_ref(other) - .downcast_ref::() - .map(|x| self.expr.eq(&x.expr) && self.order_bys.iter().eq(x.order_bys.iter())) - .unwrap_or(false) - } -} - -impl AggregateExpr for RangeFirstListValue { - fn as_any(&self) -> &dyn std::any::Any { - self - } - - fn create_accumulator(&self) -> DataFusionResult> { - Ok(Box::new(RangeFirstListValueAcc::new( - self.order_bys.iter().map(|order| order.options).collect(), - ))) - } - - fn expressions(&self) -> Vec> { - let mut exprs: Vec<_> = self - .order_bys - .iter() - .map(|order| order.expr.clone()) - .collect(); - exprs.push(self.expr.clone()); - exprs - } - - fn field(&self) -> DataFusionResult { - unreachable!("AggregateExpr::field will not be used in range query") - } - - fn state_fields(&self) -> DataFusionResult> { - unreachable!("AggregateExpr::state_fields will not be used in range query") - } -} - -#[derive(Debug)] -pub struct RangeFirstListValueAcc { - pub sort_options: Vec, - pub sort_columns: Vec, - pub data: Option, -} - -impl RangeFirstListValueAcc { - pub fn new(sort_options: Vec) -> Self { - Self { - sort_options, - sort_columns: vec![], - data: None, - } - } -} - -impl Accumulator for RangeFirstListValueAcc { - fn update_batch(&mut self, values: &[ArrayRef]) -> DataFusionResult<()> { - let columns: Vec<_> = values - .iter() - .zip(self.sort_options.iter()) - .map(|(v, s)| SortColumn { - values: v.clone(), - options: Some(*s), - }) - .collect(); - // finding the Top1 problem with complexity O(n) - let idx = compute::lexsort_to_indices(&columns, Some(1))?.value(0); - let vs = get_row_at_idx(values, idx as usize)?; - let need_update = self.data.is_none() - || vs - .iter() - .zip(self.sort_columns.iter()) - .zip(self.sort_options.iter()) - .find_map(|((new_value, old_value), sort_option)| { - if new_value.is_null() && old_value.is_null() { - None - } else if sort_option.nulls_first - && (new_value.is_null() || old_value.is_null()) - { - Some(new_value.is_null()) - } else { - new_value.partial_cmp(old_value).map(|x| { - (x == Ordering::Greater && sort_option.descending) - || (x == Ordering::Less && !sort_option.descending) - }) - } - }) - .unwrap_or(false); - if need_update { - self.sort_columns = vs; - self.data = Some(ScalarValue::try_from_array( - &values[self.sort_options.len()], - idx as usize, - )?); - } - Ok(()) - } - - fn evaluate(&mut self) -> DataFusionResult { - Ok(self.data.clone().unwrap_or(ScalarValue::Null)) - } - - fn size(&self) -> usize { - std::mem::size_of_val(self) - } - - fn state(&mut self) -> DataFusionResult> { - unreachable!("Accumulator::state will not be used in range query") - } - - fn merge_batch(&mut self, _states: &[ArrayRef]) -> DataFusionResult<()> { - unreachable!("Accumulator::merge_batch will not be used in range query") - } -} - #[derive(PartialEq, Eq, Debug, Hash, Clone)] pub enum Fill { Null, @@ -432,6 +292,41 @@ pub struct RangeSelect { pub schema_before_project: DFSchemaRef, } +impl PartialOrd for RangeSelect { + fn partial_cmp(&self, other: &Self) -> Option { + // Compare fields in order excluding `schema`, `by_schema`, `schema_before_project`. + match self.input.partial_cmp(&other.input) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.range_expr.partial_cmp(&other.range_expr) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.align.partial_cmp(&other.align) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.align_to.partial_cmp(&other.align_to) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.time_index.partial_cmp(&other.time_index) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.time_expr.partial_cmp(&other.time_expr) { + Some(Ordering::Equal) => {} + ord => return ord, + } + match self.by.partial_cmp(&other.by) { + Some(Ordering::Equal) => {} + ord => return ord, + } + self.schema_project.partial_cmp(&other.schema_project) + } +} + impl RangeSelect { pub fn try_new( input: Arc, @@ -686,7 +581,7 @@ impl RangeSelect { .range_expr .iter() .map(|range_fn| { - let name = range_fn.expr.display_name()?; + let name = range_fn.expr.schema_name().to_string(); let range_expr = match &range_fn.expr { Expr::Alias(expr) => expr.expr.as_ref(), others => others, @@ -694,16 +589,9 @@ impl RangeSelect { let expr = match &range_expr { Expr::AggregateFunction(aggr) - if (aggr.func_def.name() == "last_value" - || aggr.func_def.name() == "first_value") => + if (aggr.func.name() == "last_value" + || aggr.func.name() == "first_value") => { - let is_last_value_func = aggr.func_def.name() == "last_value"; - - // Because we only need to find the first_value/last_value, - // the complexity of sorting the entire batch is O(nlogn). - // We can sort the batch with limit 1. - // In this case, the algorithm degenerates into finding the Top1 problem with complexity O(n). - // We need reverse the sort order of last_value to correctly apply limit 1 when sorting. let order_by = if let Some(exprs) = &aggr.order_by { exprs .iter() @@ -713,20 +601,6 @@ impl RangeSelect { input_dfschema.as_ref(), session_state.execution_props(), ) - .map(|expr| { - // reverse the last_value sort - if is_last_value_func { - PhysicalSortExpr { - expr: expr.expr, - options: SortOptions { - descending: !expr.options.descending, - nulls_first: !expr.options.nulls_first, - }, - } - } else { - expr - } - }) }) .collect::>>()? } else { @@ -739,7 +613,7 @@ impl RangeSelect { vec![PhysicalSortExpr { expr: time_index, options: SortOptions { - descending: is_last_value_func, + descending: false, nulls_first: false, }, }] @@ -753,10 +627,11 @@ impl RangeSelect { // first_value/last_value has only one param. // The param have been checked by datafusion in logical plan stage. // We can safely assume that there is only one element here. - Ok(RangeFirstListValue::new_aggregate_expr( - arg[0].clone(), - order_by, - )) + AggregateExprBuilder::new(aggr.func.clone(), arg) + .schema(input_schema.clone()) + .order_by(LexOrdering::new(order_by)) + .alias(name) + .build() } Expr::AggregateFunction(aggr) => { let order_by = if let Some(exprs) = &aggr.order_by { @@ -777,45 +652,25 @@ impl RangeSelect { // TODO(discord9): add default null treatment? let input_phy_exprs = self.create_physical_expr_list( - matches!( - aggr.func_def, - AggregateFunctionDefinition::BuiltIn(AggregateFunction::Count,) - ), + aggr.func.name() == "count", &aggr.args, input_dfschema, session_state, )?; - match &aggr.func_def { - AggregateFunctionDefinition::BuiltIn(fun) => create_aggr_expr( - fun, - distinct, - &input_phy_exprs, - &order_by, - &input_schema, - name, - false, - ), - AggregateFunctionDefinition::UDF(fun) => create_aggr_udf_expr( - fun, - &input_phy_exprs, - &[], - &[], - &input_schema, - name, - false, - distinct, - ), - } + AggregateExprBuilder::new(aggr.func.clone(), input_phy_exprs) + .schema(input_schema.clone()) + .order_by(LexOrdering::new(order_by)) + .with_distinct(distinct) + .alias(name) + .build() } _ => Err(DataFusionError::Plan(format!( "Unexpected Expr: {} in RangeSelect", - range_fn.expr.canonical_name() + range_fn.expr ))), }?; - let args = expr.expressions(); Ok(RangeFnExec { - expr, - args, + expr: Arc::new(expr), range: range_fn.range.as_millis() as Millisecond, fill: range_fn.fill.clone(), need_cast: if range_fn.need_cast { @@ -855,13 +710,26 @@ impl RangeSelect { } } +/// Range function expression. #[derive(Debug, Clone)] struct RangeFnExec { - pub expr: Arc, - pub args: Vec>, - pub range: Millisecond, - pub fill: Option, - pub need_cast: Option, + expr: Arc, + range: Millisecond, + fill: Option, + need_cast: Option, +} + +impl RangeFnExec { + /// Returns the expressions to pass to the aggregator. + /// It also adds the order by expressions to the list of expressions. + /// Order-sensitive aggregators, such as `FIRST_VALUE(x ORDER BY y)` requires this. + fn expressions(&self) -> Vec> { + let mut exprs = self.expr.expressions(); + if let Some(ordering) = self.expr.order_bys() { + exprs.extend(ordering.iter().map(|sort| sort.expr.clone())); + } + exprs + } } impl Display for RangeFnExec { @@ -1009,6 +877,10 @@ impl ExecutionPlan for RangeSelectExec { fn statistics(&self) -> DataFusionResult { Ok(Statistics::new_unknown(self.schema.as_ref())) } + + fn name(&self) -> &str { + "RanegSelectExec" + } } struct RangeSelectStream { @@ -1128,7 +1000,7 @@ impl RangeSelectStream { ) })?; for i in 0..self.range_exec.len() { - let args = self.evaluate_many(&batch, &self.range_exec[i].args)?; + let args = self.evaluate_many(&batch, &self.range_exec[i].expressions())?; // use self.modify_map record (hash, align_ts) => [row_nums] produce_align_time( self.align_to, @@ -1153,7 +1025,7 @@ impl RangeSelectStream { modify_index.push((*hash, *ts, modify[0])); } let modify_rows = modify_rows.finish(); - let args = get_arrayref_at_indices(&args, &modify_rows)?; + let args = take_arrays(&args, &modify_rows, None)?; modify_index.iter().zip(offsets.windows(2)).try_for_each( |((hash, ts, row), offset)| { let (offset, length) = (offset[0], offset[1] - offset[0]); @@ -1305,6 +1177,9 @@ impl Stream for RangeSelectStream { // new batch to aggregate Some(Ok(batch)) => { if let Err(e) = self.update_range_context(batch) { + common_telemetry::debug!( + "RangeSelectStream cannot update range context, schema: {:?}, err: {:?}", self.schema, e + ); return Poll::Ready(Some(Err(e))); } } @@ -1364,14 +1239,17 @@ mod test { }; } + use std::sync::Arc; + use arrow_schema::SortOptions; use datafusion::arrow::datatypes::{ ArrowPrimitiveType, DataType, Field, Schema, TimestampMillisecondType, }; + use datafusion::functions_aggregate::min_max; use datafusion::physical_plan::memory::MemoryExec; use datafusion::physical_plan::sorts::sort::SortExec; use datafusion::prelude::SessionContext; - use datafusion_physical_expr::expressions::{self, Column}; + use datafusion_physical_expr::expressions::Column; use datafusion_physical_expr::PhysicalSortExpr; use datatypes::arrow::array::TimestampMillisecondArray; use datatypes::arrow_array::StringArray; @@ -1465,27 +1343,36 @@ mod test { Partitioning::UnknownPartitioning(1), ExecutionMode::Bounded, ); + let input_schema = memory_exec.schema().clone(); let range_select_exec = Arc::new(RangeSelectExec { input: memory_exec, range_exec: vec![ RangeFnExec { - expr: Arc::new(expressions::Min::new( - Arc::new(Column::new("value", 1)), - "MIN(value)", - data_type.clone(), - )), - args: vec![Arc::new(Column::new("value", 1))], + expr: Arc::new( + AggregateExprBuilder::new( + min_max::min_udaf(), + vec![Arc::new(Column::new("value", 1))], + ) + .schema(input_schema.clone()) + .alias("MIN(value)") + .build() + .unwrap(), + ), range: range1, fill: fill.clone(), need_cast: need_cast.clone(), }, RangeFnExec { - expr: Arc::new(expressions::Max::new( - Arc::new(Column::new("value", 1)), - "MAX(value)", - data_type, - )), - args: vec![Arc::new(Column::new("value", 1))], + expr: Arc::new( + AggregateExprBuilder::new( + min_max::max_udaf(), + vec![Arc::new(Column::new("value", 1))], + ) + .schema(input_schema.clone()) + .alias("MAX(value)") + .build() + .unwrap(), + ), range: range2, fill, need_cast, @@ -1503,7 +1390,7 @@ mod test { cache, }); let sort_exec = SortExec::new( - vec![ + LexOrdering::new(vec![ PhysicalSortExpr { expr: Arc::new(Column::new("host", 3)), options: SortOptions { @@ -1518,7 +1405,7 @@ mod test { nulls_first: true, }, }, - ], + ]), range_select_exec, ); let session_context = SessionContext::default(); @@ -1908,44 +1795,4 @@ mod test { Fill::Linear.apply_fill_strategy(&ts, &mut test1).unwrap(); assert_eq!(test, test1); } - - #[test] - fn test_fist_last_accumulator() { - let mut acc = RangeFirstListValueAcc::new(vec![ - SortOptions { - descending: true, - nulls_first: false, - }, - SortOptions { - descending: false, - nulls_first: true, - }, - ]); - let batch1: Vec> = vec![ - Arc::new(nullable_array!(Float64; - 0.0, null, 0.0, null, 1.0 - )), - Arc::new(nullable_array!(Float64; - 5.0, null, 4.0, null, 3.0 - )), - Arc::new(nullable_array!(Int64; - 1, 2, 3, 4, 5 - )), - ]; - let batch2: Vec> = vec![ - Arc::new(nullable_array!(Float64; - 3.0, 3.0, 3.0, 3.0, 3.0 - )), - Arc::new(nullable_array!(Float64; - null,3.0, 3.0, 3.0, 3.0 - )), - Arc::new(nullable_array!(Int64; - 6, 7, 8, 9, 10 - )), - ]; - acc.update_batch(&batch1).unwrap(); - assert_eq!(acc.evaluate().unwrap(), ScalarValue::Int64(Some(5))); - acc.update_batch(&batch2).unwrap(); - assert_eq!(acc.evaluate().unwrap(), ScalarValue::Int64(Some(6))); - } } diff --git a/src/query/src/range_select/plan_rewrite.rs b/src/query/src/range_select/plan_rewrite.rs index 56cad37ef2..db0c7e4050 100644 --- a/src/query/src/range_select/plan_rewrite.rs +++ b/src/query/src/range_select/plan_rewrite.rs @@ -29,6 +29,7 @@ use datafusion::scalar::ScalarValue; use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion, TreeNodeRewriter}; use datafusion_common::{DFSchema, DataFusionError, Result as DFResult}; use datafusion_expr::execution_props::ExecutionProps; +use datafusion_expr::expr::WildcardOptions; use datafusion_expr::simplify::SimplifyContext; use datafusion_expr::{ Aggregate, Analyze, Explain, Expr, ExprSchemable, Extension, LogicalPlan, LogicalPlanBuilder, @@ -46,6 +47,7 @@ use crate::error::{ CatalogSnafu, DataFusionSnafu, RangeQuerySnafu, Result, TimeIndexNotFoundSnafu, UnknownTableSnafu, }; +use crate::plan::ExtractExpr; use crate::range_select::plan::{RangeFn, RangeSelect}; /// `RangeExprRewriter` will recursively search certain `Expr`, find all `range_fn` scalar udf contained in `Expr`, @@ -87,7 +89,7 @@ fn dispose_parse_error(expr: Option<&Expr>) -> DataFusionError { expr.map(|x| { format!( "Illegal argument `{}` in range select query", - x.display_name().unwrap_or_default() + x.schema_name() ) }) .unwrap_or("Missing argument in range select query".into()), @@ -104,7 +106,7 @@ fn parse_str_expr(args: &[Expr], i: usize) -> DFResult<&str> { fn parse_expr_to_string(args: &[Expr], i: usize) -> DFResult { match args.get(i) { Some(Expr::Literal(ScalarValue::Utf8(Some(str)))) => Ok(str.to_string()), - Some(expr) => Ok(expr.display_name().unwrap_or_default()), + Some(expr) => Ok(expr.schema_name().to_string()), None => Err(dispose_parse_error(None)), } } @@ -166,7 +168,7 @@ fn evaluate_expr_to_millisecond(args: &[Expr], i: usize, interval_only: bool) -> if interval.months != 0 { return Err(DataFusionError::Plan(format!( "Year or month interval is not allowed in range query: {}", - expr.display_name().unwrap_or_default() + expr.schema_name() ))); } @@ -174,16 +176,16 @@ fn evaluate_expr_to_millisecond(args: &[Expr], i: usize, interval_only: bool) -> }) .transpose()?, Expr::Literal(ScalarValue::IntervalDayTime(interval)) => interval.map(|v| { - let interval = IntervalDayTime::from_i64(v); + let interval = IntervalDayTime::from(v); interval.as_millis() }), Expr::Literal(ScalarValue::IntervalMonthDayNano(interval)) => interval .map(|v| { - let interval = IntervalMonthDayNano::from_i128(v); + let interval = IntervalMonthDayNano::from(v); if interval.months != 0 { return Err(DataFusionError::Plan(format!( "Year or month interval is not allowed in range query: {}", - expr.display_name().unwrap_or_default() + expr.schema_name() ))); } @@ -195,7 +197,7 @@ fn evaluate_expr_to_millisecond(args: &[Expr], i: usize, interval_only: bool) -> .ok_or_else(|| { DataFusionError::Plan(format!( "{} is not a expr can be evaluate and use in range query", - expr.display_name().unwrap_or_default() + expr.schema_name() )) }) } @@ -293,14 +295,14 @@ impl TreeNodeRewriter for RangeExprRewriter<'_> { name: if let Some(fill) = &fill { format!( "{} RANGE {} FILL {}", - range_expr.display_name()?, + range_expr.schema_name(), parse_expr_to_string(&func.args, 1)?, fill ) } else { format!( "{} RANGE {}", - range_expr.display_name()?, + range_expr.schema_name(), parse_expr_to_string(&func.args, 1)?, ) }, @@ -451,7 +453,7 @@ impl RangePlanRewriter { .context(DataFusionSnafu)? .build() } - _ => plan.with_new_exprs(plan.expressions(), inputs), + _ => plan.with_new_exprs(plan.expressions_consider_join(), inputs), } .context(DataFusionSnafu)?; Ok(Some(plan)) @@ -467,7 +469,10 @@ impl RangePlanRewriter { /// If the user does not explicitly use the `by` keyword to indicate time series, /// `[row_columns]` will be use as default time series async fn get_index_by(&mut self, schema: &Arc) -> Result<(Expr, Vec)> { - let mut time_index_expr = Expr::Wildcard { qualifier: None }; + let mut time_index_expr = Expr::Wildcard { + qualifier: None, + options: WildcardOptions::default(), + }; let mut default_by = vec![]; for i in 0..schema.fields().len() { let (qualifier, _) = schema.qualified_field(i); @@ -646,7 +651,7 @@ mod test { async fn range_no_project() { let query = r#"SELECT timestamp, tag_0, tag_1, avg(field_0 + field_1) RANGE '5m' FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "RangeSelect: range_exprs=[AVG(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8, AVG(test.field_0 + test.field_1) RANGE 5m:Float64;N]\ + "RangeSelect: range_exprs=[avg(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8, avg(test.field_0 + test.field_1) RANGE 5m:Float64;N]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -656,8 +661,8 @@ mod test { async fn range_expr_calculation() { let query = r#"SELECT (avg(field_0 + field_1)/4) RANGE '5m' FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "Projection: AVG(test.field_0 + test.field_1) RANGE 5m / Int64(4) [AVG(test.field_0 + test.field_1) RANGE 5m / Int64(4):Float64;N]\ - \n RangeSelect: range_exprs=[AVG(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: avg(test.field_0 + test.field_1) RANGE 5m / Int64(4) [avg(test.field_0 + test.field_1) RANGE 5m / Int64(4):Float64;N]\ + \n RangeSelect: range_exprs=[avg(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -679,8 +684,8 @@ mod test { async fn range_calculation() { let query = r#"SELECT ((avg(field_0)+sum(field_1))/4) RANGE '5m' FROM test ALIGN '1h' by (tag_0,tag_1) FILL NULL;"#; let expected = String::from( - "Projection: (AVG(test.field_0) RANGE 5m FILL NULL + SUM(test.field_1) RANGE 5m FILL NULL) / Int64(4) [AVG(test.field_0) RANGE 5m FILL NULL + SUM(test.field_1) RANGE 5m FILL NULL / Int64(4):Float64;N]\ - \n RangeSelect: range_exprs=[AVG(test.field_0) RANGE 5m FILL NULL, SUM(test.field_1) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0) RANGE 5m FILL NULL:Float64;N, SUM(test.field_1) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: (avg(test.field_0) RANGE 5m FILL NULL + sum(test.field_1) RANGE 5m FILL NULL) / Int64(4) [avg(test.field_0) RANGE 5m FILL NULL + sum(test.field_1) RANGE 5m FILL NULL / Int64(4):Float64;N]\ + \n RangeSelect: range_exprs=[avg(test.field_0) RANGE 5m FILL NULL, sum(test.field_1) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0) RANGE 5m FILL NULL:Float64;N, sum(test.field_1) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -692,8 +697,8 @@ mod test { let expected = String::from( "Projection: foo + Int64(1) [foo + Int64(1):Float64;N]\ \n Filter: foo > Int64(1) [foo:Float64;N]\ - \n Projection: (AVG(test.field_0) RANGE 5m FILL NULL + SUM(test.field_1) RANGE 5m FILL NULL) / Int64(4) AS foo [foo:Float64;N]\ - \n RangeSelect: range_exprs=[AVG(test.field_0) RANGE 5m FILL NULL, SUM(test.field_1) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0) RANGE 5m FILL NULL:Float64;N, SUM(test.field_1) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + \n Projection: (avg(test.field_0) RANGE 5m FILL NULL + sum(test.field_1) RANGE 5m FILL NULL) / Int64(4) AS foo [foo:Float64;N]\ + \n RangeSelect: range_exprs=[avg(test.field_0) RANGE 5m FILL NULL, sum(test.field_1) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0) RANGE 5m FILL NULL:Float64;N, sum(test.field_1) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -703,8 +708,8 @@ mod test { async fn range_from_nest_query() { let query = r#"SELECT ((avg(a)+sum(b))/4) RANGE '5m' FROM (SELECT field_0 as a, field_1 as b, tag_0 as c, tag_1 as d, timestamp from test where field_0 > 1.0) ALIGN '1h' by (c, d) FILL NULL;"#; let expected = String::from( - "Projection: (AVG(a) RANGE 5m FILL NULL + SUM(b) RANGE 5m FILL NULL) / Int64(4) [AVG(a) RANGE 5m FILL NULL + SUM(b) RANGE 5m FILL NULL / Int64(4):Float64;N]\ - \n RangeSelect: range_exprs=[AVG(a) RANGE 5m FILL NULL, SUM(b) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[c, d], time_index=timestamp [AVG(a) RANGE 5m FILL NULL:Float64;N, SUM(b) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), c:Utf8, d:Utf8]\ + "Projection: (avg(a) RANGE 5m FILL NULL + sum(b) RANGE 5m FILL NULL) / Int64(4) [avg(a) RANGE 5m FILL NULL + sum(b) RANGE 5m FILL NULL / Int64(4):Float64;N]\ + \n RangeSelect: range_exprs=[avg(a) RANGE 5m FILL NULL, sum(b) RANGE 5m FILL NULL], align=3600000ms, align_to=0ms, align_by=[c, d], time_index=timestamp [avg(a) RANGE 5m FILL NULL:Float64;N, sum(b) RANGE 5m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), c:Utf8, d:Utf8]\ \n Projection: test.field_0 AS a, test.field_1 AS b, test.tag_0 AS c, test.tag_1 AS d, test.timestamp [a:Float64;N, b:Float64;N, c:Utf8, d:Utf8, timestamp:Timestamp(Millisecond, None)]\ \n Filter: test.field_0 > Float64(1) [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" @@ -716,8 +721,8 @@ mod test { async fn range_in_expr() { let query = r#"SELECT sin(avg(field_0 + field_1) RANGE '5m' + 1) FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "Projection: sin(AVG(test.field_0 + test.field_1) RANGE 5m + Int64(1)) [sin(AVG(test.field_0 + test.field_1) RANGE 5m + Int64(1)):Float64;N]\ - \n RangeSelect: range_exprs=[AVG(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: sin(avg(test.field_0 + test.field_1) RANGE 5m + Int64(1)) [sin(avg(test.field_0 + test.field_1) RANGE 5m + Int64(1)):Float64;N]\ + \n RangeSelect: range_exprs=[avg(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -727,8 +732,8 @@ mod test { async fn duplicate_range_expr() { let query = r#"SELECT avg(field_0) RANGE '5m' FILL 6.0 + avg(field_0) RANGE '5m' FILL 6.0 FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "Projection: AVG(test.field_0) RANGE 5m FILL 6 + AVG(test.field_0) RANGE 5m FILL 6 [AVG(test.field_0) RANGE 5m FILL 6 + AVG(test.field_0) RANGE 5m FILL 6:Float64]\ - \n RangeSelect: range_exprs=[AVG(test.field_0) RANGE 5m FILL 6], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0) RANGE 5m FILL 6:Float64, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: avg(test.field_0) RANGE 5m FILL 6 + avg(test.field_0) RANGE 5m FILL 6 [avg(test.field_0) RANGE 5m FILL 6 + avg(test.field_0) RANGE 5m FILL 6:Float64]\ + \n RangeSelect: range_exprs=[avg(test.field_0) RANGE 5m FILL 6], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0) RANGE 5m FILL 6:Float64, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -738,8 +743,8 @@ mod test { async fn deep_nest_range_expr() { let query = r#"SELECT round(sin(avg(field_0 + field_1) RANGE '5m' + 1)) FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "Projection: round(sin(AVG(test.field_0 + test.field_1) RANGE 5m + Int64(1))) [round(sin(AVG(test.field_0 + test.field_1) RANGE 5m + Int64(1))):Float64;N]\ - \n RangeSelect: range_exprs=[AVG(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [AVG(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: round(sin(avg(test.field_0 + test.field_1) RANGE 5m + Int64(1))) [round(sin(avg(test.field_0 + test.field_1) RANGE 5m + Int64(1))):Float64;N]\ + \n RangeSelect: range_exprs=[avg(test.field_0 + test.field_1) RANGE 5m], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [avg(test.field_0 + test.field_1) RANGE 5m:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -749,8 +754,8 @@ mod test { async fn complex_range_expr() { let query = r#"SELECT gcd(CAST(max(field_0 + 1) Range '5m' FILL NULL AS Int64), CAST(tag_0 AS Int64)) + round(max(field_2+1) Range '6m' FILL NULL + 1) + max(field_2+3) Range '10m' FILL NULL * CAST(tag_1 AS Float64) + 1 FROM test ALIGN '1h' by (tag_0, tag_1);"#; let expected = String::from( - "Projection: gcd(arrow_cast(MAX(test.field_0 + Int64(1)) RANGE 5m FILL NULL, Utf8(\"Int64\")), arrow_cast(test.tag_0, Utf8(\"Int64\"))) + round(MAX(test.field_2 + Int64(1)) RANGE 6m FILL NULL + Int64(1)) + MAX(test.field_2 + Int64(3)) RANGE 10m FILL NULL * arrow_cast(test.tag_1, Utf8(\"Float64\")) + Int64(1) [gcd(arrow_cast(MAX(test.field_0 + Int64(1)) RANGE 5m FILL NULL,Utf8(\"Int64\")),arrow_cast(test.tag_0,Utf8(\"Int64\"))) + round(MAX(test.field_2 + Int64(1)) RANGE 6m FILL NULL + Int64(1)) + MAX(test.field_2 + Int64(3)) RANGE 10m FILL NULL * arrow_cast(test.tag_1,Utf8(\"Float64\")) + Int64(1):Float64;N]\ - \n RangeSelect: range_exprs=[MAX(test.field_0 + Int64(1)) RANGE 5m FILL NULL, MAX(test.field_2 + Int64(1)) RANGE 6m FILL NULL, MAX(test.field_2 + Int64(3)) RANGE 10m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [MAX(test.field_0 + Int64(1)) RANGE 5m FILL NULL:Float64;N, MAX(test.field_2 + Int64(1)) RANGE 6m FILL NULL:Float64;N, MAX(test.field_2 + Int64(3)) RANGE 10m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ + "Projection: gcd(arrow_cast(max(test.field_0 + Int64(1)) RANGE 5m FILL NULL, Utf8(\"Int64\")), arrow_cast(test.tag_0, Utf8(\"Int64\"))) + round(max(test.field_2 + Int64(1)) RANGE 6m FILL NULL + Int64(1)) + max(test.field_2 + Int64(3)) RANGE 10m FILL NULL * arrow_cast(test.tag_1, Utf8(\"Float64\")) + Int64(1) [gcd(arrow_cast(max(test.field_0 + Int64(1)) RANGE 5m FILL NULL,Utf8(\"Int64\")),arrow_cast(test.tag_0,Utf8(\"Int64\"))) + round(max(test.field_2 + Int64(1)) RANGE 6m FILL NULL + Int64(1)) + max(test.field_2 + Int64(3)) RANGE 10m FILL NULL * arrow_cast(test.tag_1,Utf8(\"Float64\")) + Int64(1):Float64;N]\ + \n RangeSelect: range_exprs=[max(test.field_0 + Int64(1)) RANGE 5m FILL NULL, max(test.field_2 + Int64(1)) RANGE 6m FILL NULL, max(test.field_2 + Int64(3)) RANGE 10m FILL NULL], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [max(test.field_0 + Int64(1)) RANGE 5m FILL NULL:Float64;N, max(test.field_2 + Int64(1)) RANGE 6m FILL NULL:Float64;N, max(test.field_2 + Int64(3)) RANGE 10m FILL NULL:Float64;N, timestamp:Timestamp(Millisecond, None), tag_0:Utf8, tag_1:Utf8]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -760,7 +765,7 @@ mod test { async fn range_linear_on_integer() { let query = r#"SELECT min(CAST(field_0 AS Int64) + CAST(field_1 AS Int64)) RANGE '5m' FILL LINEAR FROM test ALIGN '1h' by (tag_0,tag_1);"#; let expected = String::from( - "RangeSelect: range_exprs=[MIN(arrow_cast(test.field_0,Utf8(\"Int64\")) + arrow_cast(test.field_1,Utf8(\"Int64\"))) RANGE 5m FILL LINEAR], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [MIN(arrow_cast(test.field_0,Utf8(\"Int64\")) + arrow_cast(test.field_1,Utf8(\"Int64\"))) RANGE 5m FILL LINEAR:Float64;N]\ + "RangeSelect: range_exprs=[min(arrow_cast(test.field_0,Utf8(\"Int64\")) + arrow_cast(test.field_1,Utf8(\"Int64\"))) RANGE 5m FILL LINEAR], align=3600000ms, align_to=0ms, align_by=[test.tag_0, test.tag_1], time_index=timestamp [min(arrow_cast(test.field_0,Utf8(\"Int64\")) + arrow_cast(test.field_1,Utf8(\"Int64\"))) RANGE 5m FILL LINEAR:Float64;N]\ \n TableScan: test [tag_0:Utf8, tag_1:Utf8, tag_2:Utf8, tag_3:Utf8, tag_4:Utf8, timestamp:Timestamp(Millisecond, None), field_0:Float64;N, field_1:Float64;N, field_2:Float64;N, field_3:Float64;N, field_4:Float64;N]" ); query_plan_compare(query, expected).await; @@ -818,7 +823,7 @@ mod test { // test IntervalDayTime let interval = IntervalDayTime::new(10, 10); let args = vec![Expr::Literal(ScalarValue::IntervalDayTime(Some( - interval.to_i64(), + interval.into(), )))]; assert_eq!( parse_duration_expr(&args, 0).unwrap().as_millis() as i64, @@ -827,7 +832,7 @@ mod test { // test IntervalMonthDayNano let interval = IntervalMonthDayNano::new(0, 10, 10); let args = vec![Expr::Literal(ScalarValue::IntervalMonthDayNano(Some( - interval.to_i128(), + interval.into(), )))]; assert_eq!( parse_duration_expr(&args, 0).unwrap().as_millis() as i64, @@ -844,11 +849,11 @@ mod test { // test evaluate expr let args = vec![Expr::BinaryExpr(BinaryExpr { left: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), op: Operator::Plus, right: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), })]; assert_eq!( @@ -857,11 +862,11 @@ mod test { ); let args = vec![Expr::BinaryExpr(BinaryExpr { left: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), op: Operator::Minus, right: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), })]; // test zero interval error @@ -923,11 +928,11 @@ mod test { // test evaluate expr let args = vec![Expr::BinaryExpr(BinaryExpr { left: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), op: Operator::Plus, right: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(0, 10).to_i64(), + IntervalDayTime::new(0, 10).into(), )))), })]; assert_eq!(parse_align_to(&args, 0, None).unwrap(), 20); @@ -939,17 +944,17 @@ mod test { left: Box::new(Expr::Literal(ScalarValue::DurationMillisecond(Some(20)))), op: Operator::Minus, right: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(10, 0).to_i64(), + IntervalDayTime::new(10, 0).into(), )))), }); assert!(!interval_only_in_expr(&expr)); let expr = Expr::BinaryExpr(BinaryExpr { left: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(10, 0).to_i64(), + IntervalDayTime::new(10, 0).into(), )))), op: Operator::Minus, right: Box::new(Expr::Literal(ScalarValue::IntervalDayTime(Some( - IntervalDayTime::new(10, 0).to_i64(), + IntervalDayTime::new(10, 0).into(), )))), }); assert!(interval_only_in_expr(&expr)); diff --git a/src/query/src/sql.rs b/src/query/src/sql.rs index 3fe46b486d..5e262b7639 100644 --- a/src/query/src/sql.rs +++ b/src/query/src/sql.rs @@ -41,7 +41,8 @@ use common_time::timezone::get_timezone; use common_time::Timestamp; use datafusion::common::ScalarValue; use datafusion::prelude::{concat_ws, SessionContext}; -use datafusion_expr::{case, col, lit, Expr}; +use datafusion_expr::expr::WildcardOptions; +use datafusion_expr::{case, col, lit, Expr, SortExpr}; use datatypes::prelude::*; use datatypes::schema::{ColumnDefaultConstraint, ColumnSchema, RawSchema, Schema}; use datatypes::vectors::StringVector; @@ -227,7 +228,7 @@ async fn query_from_information_schema_table( projects: Vec<(&str, &str)>, filters: Vec, like_field: Option<&str>, - sort: Vec, + sort: Vec, kind: ShowKind, ) -> Result { let table = catalog_manager @@ -450,7 +451,10 @@ pub async fn show_index( lit("").alias(INDEX_COMMENT_COLUMN), lit(YES_STR).alias(INDEX_VISIBLE_COLUMN), null().alias(INDEX_EXPRESSION_COLUMN), - Expr::Wildcard { qualifier: None }, + Expr::Wildcard { + qualifier: None, + options: WildcardOptions::default(), + }, ]; let projects = vec![ diff --git a/src/query/src/test_util.rs b/src/query/src/test_util.rs index 77a2dee1de..bec2099da3 100644 --- a/src/query/src/test_util.rs +++ b/src/query/src/test_util.rs @@ -73,6 +73,10 @@ impl DisplayAs for MockInputExec { } impl ExecutionPlan for MockInputExec { + fn name(&self) -> &str { + "MockInputExec" + } + fn as_any(&self) -> &dyn Any { self } diff --git a/src/query/src/window_sort.rs b/src/query/src/window_sort.rs index c1d5ea430b..be8fe52549 100644 --- a/src/query/src/window_sort.rs +++ b/src/query/src/window_sort.rs @@ -33,13 +33,13 @@ use datafusion::execution::memory_pool::{MemoryConsumer, MemoryPool}; use datafusion::execution::{RecordBatchStream, TaskContext}; use datafusion::physical_plan::memory::MemoryStream; use datafusion::physical_plan::metrics::{BaselineMetrics, ExecutionPlanMetricsSet, MetricsSet}; -use datafusion::physical_plan::sorts::streaming_merge::streaming_merge; +use datafusion::physical_plan::sorts::streaming_merge::StreamingMergeBuilder; use datafusion::physical_plan::{ DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, PlanProperties, }; use datafusion_common::utils::bisect; use datafusion_common::{internal_err, DataFusionError}; -use datafusion_physical_expr::PhysicalSortExpr; +use datafusion_physical_expr::{LexOrdering, PhysicalSortExpr}; use datatypes::value::Value; use futures::Stream; use itertools::Itertools; @@ -52,7 +52,7 @@ use crate::error::{QueryExecutionSnafu, Result}; /// merge sort them whenever possible, and emit the sorted result as soon as possible. /// This sorting plan only accept sort by ts and will not sort by other fields. /// -/// internally, it call [`streaming_merge`] multiple times to merge multiple sorted "working ranges" +/// internally, it call [`StreamingMergeBuilder`] multiple times to merge multiple sorted "working ranges" /// /// # Invariant Promise on Input Stream /// 1. The input stream must be sorted by timestamp and @@ -121,7 +121,7 @@ impl WindowedSortExec { input .equivalence_properties() .clone() - .with_reorder(vec![expression.clone()]), + .with_reorder(LexOrdering::new(vec![expression.clone()])), input.output_partitioning().clone(), input.execution_mode(), ); @@ -237,6 +237,10 @@ impl ExecutionPlan for WindowedSortExec { fn benefits_from_input_partitioning(&self) -> Vec { vec![false; self.ranges.len()] } + + fn name(&self) -> &str { + "WindowedSortExec" + } } /// The core logic of merging sort multiple sorted ranges @@ -635,16 +639,17 @@ impl WindowedSortStream { let reservation = MemoryConsumer::new(format!("WindowedSortStream[{}]", self.merge_count)) .register(&self.memory_pool); self.merge_count += 1; + let lex_ordering = LexOrdering::new(vec![self.expression.clone()]); - let resulting_stream = streaming_merge( - streams, - self.schema(), - &[self.expression.clone()], - self.metrics.clone(), - self.batch_size, - fetch, - reservation, - )?; + let resulting_stream = StreamingMergeBuilder::new() + .with_streams(streams) + .with_schema(self.schema()) + .with_expressions(&lex_ordering) + .with_metrics(self.metrics.clone()) + .with_batch_size(self.batch_size) + .with_fetch(fetch) + .with_reservation(reservation) + .build()?; self.merge_stream.push_back(resulting_stream); // this working range is done, move to next working range Ok(()) @@ -3184,8 +3189,8 @@ mod test { .unwrap_or_else(|| rng.i64(..)); bound_val = Some(end); let start = end - rng.i64(1..range_size_bound); - let start = Timestamp::new(start, unit.clone().into()); - let end = Timestamp::new(end, unit.clone().into()); + let start = Timestamp::new(start, unit.into()); + let end = Timestamp::new(end, unit.into()); (start, end) } else { let start = bound_val @@ -3193,8 +3198,8 @@ mod test { .unwrap_or_else(|| rng.i64(..)); bound_val = Some(start); let end = start + rng.i64(1..range_size_bound); - let start = Timestamp::new(start, unit.clone().into()); - let end = Timestamp::new(end, unit.clone().into()); + let start = Timestamp::new(start, unit.into()); + let end = Timestamp::new(end, unit.into()); (start, end) }; @@ -3204,7 +3209,7 @@ mod test { .sorted_by(ret_cmp_fn(descending)) .collect_vec(); output_data.extend(data_gen.clone()); - let arr = new_ts_array(unit.clone(), data_gen); + let arr = new_ts_array(unit, data_gen); let range = PartitionRange { start, end, @@ -3218,7 +3223,7 @@ mod test { if let Some(fetch) = fetch { output_data.truncate(fetch); } - let output_arr = new_ts_array(unit.clone(), output_data); + let output_arr = new_ts_array(unit, output_data); let test_stream = TestStream::new( Column::new("ts", 0), @@ -3227,11 +3232,7 @@ mod test { nulls_first: true, }, fetch, - vec![Field::new( - "ts", - DataType::Timestamp(unit.clone(), None), - false, - )], + vec![Field::new("ts", DataType::Timestamp(unit, None), false)], input_ranged_data.clone(), vec![vec![output_arr]], ); diff --git a/src/servers/Cargo.toml b/src/servers/Cargo.toml index 5a7d3968f8..650eb94df3 100644 --- a/src/servers/Cargo.toml +++ b/src/servers/Cargo.toml @@ -27,7 +27,8 @@ arrow-schema.workspace = true async-trait = "0.1" auth.workspace = true axum = { workspace = true, features = ["multipart"] } -axum-macros = "0.3.8" +axum-extra = { workspace = true, features = ["typed-header"] } +axum-macros.workspace = true base64.workspace = true bytes.workspace = true catalog.workspace = true @@ -57,19 +58,19 @@ derive_builder.workspace = true futures = "0.3" futures-util.workspace = true hashbrown = "0.15" -headers = "0.3" +headers = "0.4" hostname = "0.3" http.workspace = true -http-body = "0.4" +http-body = "1" humantime.workspace = true humantime-serde.workspace = true -hyper = { version = "0.14", features = ["full"] } +hyper = { workspace = true, features = ["full"] } influxdb_line_protocol = { git = "https://github.com/evenyag/influxdb_iox", branch = "feat/line-protocol" } itertools.workspace = true jsonb.workspace = true lazy_static.workspace = true log-query.workspace = true -loki-api = "0.1" +loki-api.workspace = true mime_guess = "2.0" notify.workspace = true object-pool = "0.5" @@ -110,9 +111,9 @@ tokio-rustls.workspace = true tokio-stream = { workspace = true, features = ["net"] } tokio-util.workspace = true tonic.workspace = true -tonic-reflection = "0.11" +tonic-reflection = "0.12" tower = { workspace = true, features = ["full"] } -tower-http = { version = "0.4", features = ["full"] } +tower-http = { version = "0.5", features = ["full"] } urlencoding = "2.1" uuid.workspace = true zstd.workspace = true diff --git a/src/servers/src/elasticsearch.rs b/src/servers/src/elasticsearch.rs index 58c6aa520a..48638de226 100644 --- a/src/servers/src/elasticsearch.rs +++ b/src/servers/src/elasticsearch.rs @@ -16,12 +16,13 @@ use std::sync::Arc; use std::time::Instant; use axum::extract::{Path, Query, State}; -use axum::headers::ContentType; use axum::http::{HeaderMap, HeaderName, HeaderValue, StatusCode}; use axum::response::IntoResponse; -use axum::{Extension, TypedHeader}; +use axum::Extension; +use axum_extra::TypedHeader; use common_error::ext::ErrorExt; use common_telemetry::{debug, error}; +use headers::ContentType; use once_cell::sync::Lazy; use serde_json::{json, Deserializer, Value}; use session::context::{Channel, QueryContext}; diff --git a/src/servers/src/error.rs b/src/servers/src/error.rs index b8882b0b72..235195fa02 100644 --- a/src/servers/src/error.rs +++ b/src/servers/src/error.rs @@ -39,7 +39,7 @@ pub enum Error { AddressBind { addr: SocketAddr, #[snafu(source)] - error: hyper::Error, + error: std::io::Error, #[snafu(implicit)] location: Location, }, @@ -203,12 +203,6 @@ pub enum Error { location: Location, }, - #[snafu(display("Hyper error"))] - Hyper { - #[snafu(source)] - error: hyper::Error, - }, - #[snafu(display("Invalid OpenTSDB Json request"))] InvalidOpentsdbJsonRequest { #[snafu(source)] @@ -595,7 +589,7 @@ pub enum Error { }, } -pub type Result = std::result::Result; +pub type Result = std::result::Result; impl ErrorExt for Error { fn status_code(&self) -> StatusCode { @@ -668,7 +662,6 @@ impl ErrorExt for Error { Catalog { source, .. } => source.status_code(), RowWriter { source, .. } => source.status_code(), - Hyper { .. } => StatusCode::Unknown, TlsRequired { .. } => StatusCode::Unknown, Auth { source, .. } => source.status_code(), DescribeStatement { source } => source.status_code(), diff --git a/src/servers/src/grpc.rs b/src/servers/src/grpc.rs index 795884d492..3137ec07ee 100644 --- a/src/servers/src/grpc.rs +++ b/src/servers/src/grpc.rs @@ -38,7 +38,8 @@ use snafu::{ensure, OptionExt, ResultExt}; use tokio::net::TcpListener; use tokio::sync::oneshot::{self, Receiver, Sender}; use tokio::sync::Mutex; -use tonic::transport::server::{Routes, TcpIncoming}; +use tonic::service::Routes; +use tonic::transport::server::TcpIncoming; use tonic::transport::ServerTlsConfig; use tonic::{Request, Response, Status}; use tonic_reflection::server::{ServerReflection, ServerReflectionServer}; @@ -169,7 +170,10 @@ impl GrpcServer { .with_service_name("greptime.v1.GreptimeDatabase") .with_service_name("greptime.v1.HealthCheck") .with_service_name("greptime.v1.RegionServer") - .build() + .build_v1() + .inspect_err(|e| { + common_telemetry::error!(e; "Failed to build gRPC reflection server"); + }) .unwrap() } diff --git a/src/servers/src/grpc/authorize.rs b/src/servers/src/grpc/authorize.rs index c9c7644e00..ab1eb1cd8c 100644 --- a/src/servers/src/grpc/authorize.rs +++ b/src/servers/src/grpc/authorize.rs @@ -17,7 +17,6 @@ use std::result::Result as StdResult; use std::task::{Context, Poll}; use auth::UserProviderRef; -use hyper::Body; use session::context::{Channel, QueryContext}; use tonic::body::BoxBody; use tonic::server::NamedService; @@ -67,9 +66,9 @@ where type BoxFuture<'a, T> = Pin + Send + 'a>>; -impl Service> for AuthMiddleware +impl Service> for AuthMiddleware where - S: Service, Response = hyper::Response> + Clone + Send + 'static, + S: Service, Response = http::Response> + Clone + Send + 'static, S::Future: Send + 'static, { type Response = S::Response; @@ -80,7 +79,7 @@ where self.inner.poll_ready(cx) } - fn call(&mut self, mut req: hyper::Request) -> Self::Future { + fn call(&mut self, mut req: http::Request) -> Self::Future { // This is necessary because tonic internally uses `tower::buffer::Buffer`. // See https://github.com/tower-rs/tower/issues/547#issuecomment-767629149 // for details on why this is necessary. @@ -91,7 +90,7 @@ where Box::pin(async move { if let Err(status) = do_auth(&mut req, user_provider).await { - return Ok(status.to_http()); + return Ok(status.into_http()); } inner.call(req).await }) @@ -99,7 +98,7 @@ where } async fn do_auth( - req: &mut hyper::Request, + req: &mut http::Request, user_provider: Option, ) -> Result<(), tonic::Status> { let (catalog, schema) = extract_catalog_and_schema(req); @@ -137,7 +136,7 @@ mod tests { use base64::engine::general_purpose::STANDARD; use base64::Engine; use headers::Header; - use hyper::{Body, Request}; + use hyper::Request; use session::context::QueryContext; use crate::grpc::authorize::do_auth; @@ -149,7 +148,7 @@ mod tests { // auth success let authorization_val = format!("Basic {}", STANDARD.encode("greptime:greptime")); - let mut req = Request::new(Body::empty()); + let mut req = Request::new(()); req.headers_mut() .insert("authorization", authorization_val.parse().unwrap()); @@ -160,7 +159,7 @@ mod tests { // auth failed, err: user not exist. let authorization_val = format!("Basic {}", STANDARD.encode("greptime2:greptime2")); - let mut req = Request::new(Body::empty()); + let mut req = Request::new(()); req.headers_mut() .insert("authorization", authorization_val.parse().unwrap()); @@ -170,19 +169,19 @@ mod tests { #[tokio::test] async fn test_do_auth_without_user_provider() { - let mut req = Request::new(Body::empty()); + let mut req = Request::new(()); req.headers_mut() .insert("authentication", "pwd".parse().unwrap()); let auth_result = do_auth(&mut req, None).await; assert!(auth_result.is_ok()); check_req(&req, "greptime", "public", "greptime"); - let mut req = Request::new(Body::empty()); + let mut req = Request::new(()); let auth_result = do_auth(&mut req, None).await; assert!(auth_result.is_ok()); check_req(&req, "greptime", "public", "greptime"); - let mut req = Request::new(Body::empty()); + let mut req = Request::new(()); req.headers_mut() .insert(GreptimeDbName::name(), "catalog-schema".parse().unwrap()); let auth_result = do_auth(&mut req, None).await; diff --git a/src/servers/src/grpc/builder.rs b/src/servers/src/grpc/builder.rs index c2481c2213..6c2c04635e 100644 --- a/src/servers/src/grpc/builder.rs +++ b/src/servers/src/grpc/builder.rs @@ -24,7 +24,7 @@ use opentelemetry_proto::tonic::collector::trace::v1::trace_service_server::Trac use snafu::ResultExt; use tokio::sync::Mutex; use tonic::codec::CompressionEncoding; -use tonic::transport::server::RoutesBuilder; +use tonic::service::RoutesBuilder; use tonic::transport::{Identity, ServerTlsConfig}; use tower::ServiceBuilder; diff --git a/src/servers/src/http.rs b/src/servers/src/http.rs index 976120e215..8ceccda226 100644 --- a/src/servers/src/http.rs +++ b/src/servers/src/http.rs @@ -20,14 +20,13 @@ use std::time::Duration; use async_trait::async_trait; use auth::UserProviderRef; -use axum::error_handling::HandleErrorLayer; use axum::extract::DefaultBodyLimit; use axum::http::StatusCode as HttpStatusCode; -use axum::response::{IntoResponse, Json, Response}; -use axum::{middleware, routing, BoxError, Router}; +use axum::response::{IntoResponse, Response}; +use axum::serve::ListenerExt; +use axum::{middleware, routing, Router}; use common_base::readable_size::ReadableSize; use common_base::Plugins; -use common_error::status_code::StatusCode; use common_recordbatch::RecordBatch; use common_telemetry::{error, info}; use common_time::timestamp::TimeUnit; @@ -50,7 +49,9 @@ use self::authorize::AuthState; use self::result::table_result::TableResponse; use crate::configurator::ConfiguratorRef; use crate::elasticsearch; -use crate::error::{AddressBindSnafu, AlreadyStartedSnafu, Error, HyperSnafu, Result, ToJsonSnafu}; +use crate::error::{ + AddressBindSnafu, AlreadyStartedSnafu, Error, InternalIoSnafu, Result, ToJsonSnafu, +}; use crate::http::influxdb::{influxdb_health, influxdb_ping, influxdb_write_v1, influxdb_write_v2}; use crate::http::prometheus::{ build_info_query, format_query, instant_query, label_values_query, labels_query, parse_query, @@ -579,7 +580,7 @@ impl HttpServerBuilder { pub fn with_metrics_handler(self, handler: MetricsHandler) -> Self { Self { - router: self.router.nest("", HttpServer::route_metrics(handler)), + router: self.router.merge(HttpServer::route_metrics(handler)), ..self } } @@ -630,14 +631,14 @@ impl HttpServerBuilder { }); Self { - router: self.router.nest("", config_router), + router: self.router.merge(config_router), ..self } } pub fn with_extra_router(self, router: Router) -> Self { Self { - router: self.router.nest("", router), + router: self.router.merge(router), ..self } } @@ -654,6 +655,7 @@ impl HttpServerBuilder { } impl HttpServer { + /// Gets the router and adds necessary root routes (health, status, dashboard). pub fn make_app(&self) -> Router { let mut router = { let router = self.router.lock().unwrap(); @@ -689,7 +691,7 @@ impl HttpServer { ); // "/dashboard" and "/dashboard/" are two different paths in Axum. - // We cannot nest "/dashboard/", because we already mapping "/dashboard/*x" while nesting "/dashboard". + // We cannot nest "/dashboard/", because we already mapping "/dashboard/{*x}" while nesting "/dashboard". // So we explicitly route "/dashboard/" here. router = router .route( @@ -697,7 +699,7 @@ impl HttpServer { routing::get(dashboard::static_handler).post(dashboard::static_handler), ) .route( - "/dashboard/*x", + "/dashboard/{*x}", routing::get(dashboard::static_handler).post(dashboard::static_handler), ); } @@ -709,6 +711,8 @@ impl HttpServer { router } + /// Attaches middlewares and debug routes to the router. + /// Callers should call this method after [HttpServer::make_app()]. pub fn build(&self, router: Router) -> Router { let timeout_layer = if self.options.timeout != Duration::default() { Some(ServiceBuilder::new().layer(DynamicTimeoutLayer::new(self.options.timeout))) @@ -730,7 +734,6 @@ impl HttpServer { // middlewares .layer( ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) // disable on failure tracing. because printing out isn't very helpful, // and we have impl IntoResponse for Error. It will print out more detailed error messages .layer(TraceLayer::new_for_http().on_failure(())) @@ -769,7 +772,6 @@ impl HttpServer { .route("/api/v1/push", routing::post(loki::loki_ingest)) .layer( ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) .layer(RequestDecompressionLayer::new().pass_through_unaccepted(true)), ) .with_state(log_state) @@ -788,12 +790,12 @@ impl HttpServer { .route("/_license", routing::get(elasticsearch::handle_get_license)) .route("/_bulk", routing::post(elasticsearch::handle_bulk_api)) .route( - "/:index/_bulk", + "/{index}/_bulk", routing::post(elasticsearch::handle_bulk_api_with_index), ) // Return fake response for Elasticsearch ilm request. .route( - "/_ilm/policy/*path", + "/_ilm/policy/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), @@ -802,7 +804,7 @@ impl HttpServer { ) // Return fake response for Elasticsearch index template request. .route( - "/_index_template/*path", + "/_index_template/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), @@ -812,7 +814,7 @@ impl HttpServer { // Return fake response for Elasticsearch ingest pipeline request. // See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/put-pipeline-api.html. .route( - "/_ingest/*path", + "/_ingest/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), @@ -822,7 +824,7 @@ impl HttpServer { // Return fake response for Elasticsearch nodes discovery request. // See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/cluster.html. .route( - "/_nodes/*path", + "/_nodes/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), @@ -832,7 +834,7 @@ impl HttpServer { // Return fake response for Logstash APIs requests. // See: https://www.elastic.co/guide/en/elasticsearch/reference/8.8/logstash-apis.html .route( - "/logstash/*path", + "/logstash/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), @@ -840,18 +842,14 @@ impl HttpServer { )), ) .route( - "/_logstash/*path", + "/_logstash/{*path}", routing::any(( HttpStatusCode::OK, elasticsearch::elasticsearch_headers(), axum::Json(serde_json::json!({})), )), ) - .layer( - ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) - .layer(RequestDecompressionLayer::new()), - ) + .layer(ServiceBuilder::new().layer(RequestDecompressionLayer::new())) .with_state(log_state) } @@ -859,17 +857,16 @@ impl HttpServer { Router::new() .route("/logs", routing::post(event::log_ingester)) .route( - "/pipelines/:pipeline_name", + "/pipelines/{pipeline_name}", routing::post(event::add_pipeline), ) .route( - "/pipelines/:pipeline_name", + "/pipelines/{pipeline_name}", routing::delete(event::delete_pipeline), ) .route("/pipelines/dryrun", routing::post(event::pipeline_dryrun)) .layer( ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) .layer(RequestDecompressionLayer::new().pass_through_unaccepted(true)), ) .with_state(log_state) @@ -911,7 +908,7 @@ impl HttpServer { .route("/series", routing::post(series_query).get(series_query)) .route("/parse_query", routing::post(parse_query).get(parse_query)) .route( - "/label/:label_name/values", + "/label/{label_name}/values", routing::get(label_values_query), ) .with_state(prometheus_handler) @@ -960,7 +957,6 @@ impl HttpServer { .route("/api/v2/write", routing::post(influxdb_write_v2)) .layer( ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) .layer(RequestDecompressionLayer::new().pass_through_unaccepted(true)), ) .route("/ping", routing::get(influxdb_ping)) @@ -981,7 +977,6 @@ impl HttpServer { .route("/v1/logs", routing::post(otlp::logs)) .layer( ServiceBuilder::new() - .layer(HandleErrorLayer::new(handle_error)) .layer(RequestDecompressionLayer::new().pass_through_unaccepted(true)), ) .with_state(otlp_handler) @@ -1012,7 +1007,7 @@ impl Server for HttpServer { async fn start(&self, listening: SocketAddr) -> Result { let (tx, rx) = oneshot::channel(); - let server = { + let serve = { let mut shutdown_tx = self.shutdown_tx.lock().await; ensure!( shutdown_tx.is_none(), @@ -1024,30 +1019,44 @@ impl Server for HttpServer { app = configurator.config_http(app); } let app = self.build(app); - let server = axum::Server::try_bind(&listening) - .with_context(|_| AddressBindSnafu { addr: listening })? - .tcp_nodelay(true) - // Enable TCP keepalive to close the dangling established connections. - // It's configured to let the keepalive probes first send after the connection sits - // idle for 59 minutes, and then send every 10 seconds for 6 times. - // So the connection will be closed after roughly 1 hour. - .tcp_keepalive(Some(Duration::from_secs(59 * 60))) - .tcp_keepalive_interval(Some(Duration::from_secs(10))) - .tcp_keepalive_retries(Some(6)) - .serve(app.into_make_service()); + let listener = tokio::net::TcpListener::bind(listening) + .await + .context(AddressBindSnafu { addr: listening })? + .tap_io(|tcp_stream| { + if let Err(e) = tcp_stream.set_nodelay(true) { + error!(e; "Failed to set TCP_NODELAY on incoming connection"); + } + }); + let serve = axum::serve(listener, app.into_make_service()); + + // FIXME(yingwen): Support keepalive. + // See: + // - https://github.com/tokio-rs/axum/discussions/2939 + // - https://stackoverflow.com/questions/73069718/how-do-i-keep-alive-tokiotcpstream-in-rust + // let server = axum::Server::try_bind(&listening) + // .with_context(|_| AddressBindSnafu { addr: listening })? + // .tcp_nodelay(true) + // // Enable TCP keepalive to close the dangling established connections. + // // It's configured to let the keepalive probes first send after the connection sits + // // idle for 59 minutes, and then send every 10 seconds for 6 times. + // // So the connection will be closed after roughly 1 hour. + // .tcp_keepalive(Some(Duration::from_secs(59 * 60))) + // .tcp_keepalive_interval(Some(Duration::from_secs(10))) + // .tcp_keepalive_retries(Some(6)) + // .serve(app.into_make_service()); *shutdown_tx = Some(tx); - server + serve }; - let listening = server.local_addr(); + let listening = serve.local_addr().context(InternalIoSnafu)?; info!("HTTP server is bound to {}", listening); common_runtime::spawn_global(async move { - if let Err(e) = server + if let Err(e) = serve .with_graceful_shutdown(rx.map(drop)) .await - .context(HyperSnafu) + .context(InternalIoSnafu) { error!(e; "Failed to shutdown http server"); } @@ -1060,15 +1069,6 @@ impl Server for HttpServer { } } -/// handle error middleware -async fn handle_error(err: BoxError) -> Json { - error!(err; "Unhandled internal error: {}", err.to_string()); - Json(HttpResponse::Error(ErrorResponse::from_error_message( - StatusCode::Unexpected, - format!("Unhandled internal error: {err}"), - ))) -} - #[cfg(test)] mod test { use std::future::pending; @@ -1170,13 +1170,7 @@ mod test { .build(); server.build(server.make_app()).route( "/test/timeout", - get(forever.layer( - ServiceBuilder::new() - .layer(HandleErrorLayer::new(|_: BoxError| async { - StatusCode::REQUEST_TIMEOUT - })) - .layer(timeout()), - )), + get(forever.layer(ServiceBuilder::new().layer(timeout()))), ) } @@ -1189,9 +1183,11 @@ mod test { #[tokio::test] async fn test_http_server_request_timeout() { + common_telemetry::init_default_ut_logging(); + let (tx, _rx) = mpsc::channel(100); let app = make_test_app(tx); - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res = client.get("/test/timeout").send().await; assert_eq!(res.status(), StatusCode::REQUEST_TIMEOUT); diff --git a/src/servers/src/http/authorize.rs b/src/servers/src/http/authorize.rs index d7bacf04c3..f04f1676f0 100644 --- a/src/servers/src/http/authorize.rs +++ b/src/servers/src/http/authorize.rs @@ -13,8 +13,8 @@ // limitations under the License. use ::auth::UserProviderRef; -use axum::extract::State; -use axum::http::{self, Request, StatusCode}; +use axum::extract::{Request, State}; +use axum::http::{self, StatusCode}; use axum::middleware::Next; use axum::response::{IntoResponse, Response}; use base64::prelude::BASE64_STANDARD; @@ -30,14 +30,13 @@ use headers::Header; use session::context::QueryContextBuilder; use snafu::{ensure, OptionExt, ResultExt}; -use super::header::{GreptimeDbName, GREPTIME_TIMEZONE_HEADER_NAME}; -use super::{AUTHORIZATION_HEADER, PUBLIC_APIS}; use crate::error::{ self, InvalidAuthHeaderInvisibleASCIISnafu, InvalidAuthHeaderSnafu, InvalidParameterSnafu, NotFoundInfluxAuthSnafu, Result, UnsupportedAuthSchemeSnafu, UrlDecodeSnafu, }; +use crate::http::header::{GreptimeDbName, GREPTIME_TIMEZONE_HEADER_NAME}; use crate::http::result::error_result::ErrorResponse; -use crate::http::HTTP_API_PREFIX; +use crate::http::{AUTHORIZATION_HEADER, HTTP_API_PREFIX, PUBLIC_APIS}; use crate::influxdb::{is_influxdb_request, is_influxdb_v2_request}; /// AuthState is a holder state for [`UserProviderRef`] @@ -115,10 +114,10 @@ pub async fn inner_auth( } } -pub async fn check_http_auth( +pub async fn check_http_auth( State(auth_state): State, - req: Request, - next: Next, + req: Request, + next: Next, ) -> Response { match inner_auth(auth_state.user_provider, req).await { Ok(req) => next.run(req).await, diff --git a/src/servers/src/http/dashboard.rs b/src/servers/src/http/dashboard.rs index 36ec2f9b30..bd02b7f37d 100644 --- a/src/servers/src/http/dashboard.rs +++ b/src/servers/src/http/dashboard.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use axum::body::{boxed, Full}; +use axum::body::Body; use axum::http::{header, StatusCode, Uri}; use axum::response::Response; use common_telemetry::debug; @@ -48,7 +48,7 @@ fn index_page() -> Result { fn get_assets(path: &str) -> Result { match Assets::get(path) { Some(content) => { - let body = boxed(Full::from(content.data)); + let body = Body::from(content.data); let mime = mime_guess::from_path(path).first_or_octet_stream(); Response::builder() @@ -57,7 +57,7 @@ fn get_assets(path: &str) -> Result { } None => Response::builder() .status(StatusCode::NOT_FOUND) - .body(boxed(Full::from("404"))), + .body(Body::from("404")), } .context(BuildHttpResponseSnafu) } diff --git a/src/servers/src/http/event.rs b/src/servers/src/http/event.rs index 17fc56f561..f77220d838 100644 --- a/src/servers/src/http/event.rs +++ b/src/servers/src/http/event.rs @@ -12,23 +12,23 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::result::Result as StdResult; use std::str::FromStr; use std::sync::Arc; use std::time::Instant; use api::v1::{RowInsertRequest, RowInsertRequests, Rows}; -use axum::body::HttpBody; -use axum::extract::{FromRequest, Multipart, Path, Query, State}; -use axum::headers::ContentType; +use async_trait::async_trait; +use axum::extract::{FromRequest, Multipart, Path, Query, Request, State}; use axum::http::header::CONTENT_TYPE; -use axum::http::{Request, StatusCode}; +use axum::http::StatusCode; use axum::response::{IntoResponse, Response}; -use axum::{async_trait, BoxError, Extension, Json, TypedHeader}; +use axum::{Extension, Json}; +use axum_extra::TypedHeader; use common_error::ext::ErrorExt; use common_query::{Output, OutputData}; use common_telemetry::{error, warn}; use datatypes::value::column_data_to_json; +use headers::ContentType; use lazy_static::lazy_static; use pipeline::error::PipelineTransformSnafu; use pipeline::util::to_pipeline_version; @@ -96,18 +96,13 @@ pub(crate) struct LogIngestRequest { pub struct PipelineContent(String); -#[async_trait] -impl FromRequest for PipelineContent +impl FromRequest for PipelineContent where - B: HttpBody + Send + 'static, - B::Data: Send, - bytes::Bytes: std::convert::From<::Data>, - B::Error: Into, S: Send + Sync, { type Rejection = Response; - async fn from_request(req: Request, state: &S) -> StdResult { + async fn from_request(req: Request, state: &S) -> Result { let content_type_header = req.headers().get(CONTENT_TYPE); let content_type = content_type_header.and_then(|value| value.to_str().ok()); if let Some(content_type) = content_type { @@ -237,7 +232,7 @@ pub async fn delete_pipeline( /// Transform NDJSON array into a single array /// always return an array fn transform_ndjson_array_factory( - values: impl IntoIterator>, + values: impl IntoIterator>, ignore_error: bool, ) -> Result> { values diff --git a/src/servers/src/http/extractor.rs b/src/servers/src/http/extractor.rs index 3a9bddcd8f..f3ae606636 100644 --- a/src/servers/src/http/extractor.rs +++ b/src/servers/src/http/extractor.rs @@ -13,9 +13,7 @@ // limitations under the License. use core::str; -use std::result::Result as StdResult; -use axum::async_trait; use axum::extract::FromRequestParts; use axum::http::request::Parts; use axum::http::StatusCode; @@ -32,14 +30,13 @@ use crate::http::header::constants::{ /// using [`GREPTIME_LOG_TABLE_NAME_HEADER_NAME`] as key. pub struct LogTableName(pub Option); -#[async_trait] impl FromRequestParts for LogTableName where S: Send + Sync, { type Rejection = (StatusCode, String); - async fn from_request_parts(parts: &mut Parts, _state: &S) -> StdResult { + async fn from_request_parts(parts: &mut Parts, _state: &S) -> Result { let headers = &parts.headers; string_value_from_header(headers, GREPTIME_LOG_TABLE_NAME_HEADER_NAME).map(LogTableName) } @@ -49,14 +46,13 @@ where /// using [`GREPTIME_TRACE_TABLE_NAME_HEADER_NAME`] as key. pub struct TraceTableName(pub Option); -#[async_trait] impl FromRequestParts for TraceTableName where S: Send + Sync, { type Rejection = (StatusCode, String); - async fn from_request_parts(parts: &mut Parts, _state: &S) -> StdResult { + async fn from_request_parts(parts: &mut Parts, _state: &S) -> Result { let headers = &parts.headers; string_value_from_header(headers, GREPTIME_TRACE_TABLE_NAME_HEADER_NAME).map(TraceTableName) } @@ -67,14 +63,13 @@ where /// See [`SelectInfo`] for more details. pub struct SelectInfoWrapper(pub SelectInfo); -#[async_trait] impl FromRequestParts for SelectInfoWrapper where S: Send + Sync, { type Rejection = (StatusCode, String); - async fn from_request_parts(parts: &mut Parts, _state: &S) -> StdResult { + async fn from_request_parts(parts: &mut Parts, _state: &S) -> Result { let select = string_value_from_header(&parts.headers, GREPTIME_LOG_EXTRACT_KEYS_HEADER_NAME)?; @@ -98,14 +93,13 @@ pub struct PipelineInfo { pub pipeline_version: Option, } -#[async_trait] impl FromRequestParts for PipelineInfo where S: Send + Sync, { type Rejection = (StatusCode, String); - async fn from_request_parts(parts: &mut Parts, _state: &S) -> StdResult { + async fn from_request_parts(parts: &mut Parts, _state: &S) -> Result { let headers = &parts.headers; let pipeline_name = string_value_from_header(headers, GREPTIME_LOG_PIPELINE_NAME_HEADER_NAME)?; @@ -132,7 +126,7 @@ where fn string_value_from_header( headers: &HeaderMap, header_key: &str, -) -> StdResult, (StatusCode, String)> { +) -> Result, (StatusCode, String)> { headers .get(header_key) .map(|value| { diff --git a/src/servers/src/http/hints.rs b/src/servers/src/http/hints.rs index 4612201880..7f98461cf6 100644 --- a/src/servers/src/http/hints.rs +++ b/src/servers/src/http/hints.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use axum::body::Body; use axum::http::Request; use axum::middleware::Next; use axum::response::Response; @@ -19,7 +20,7 @@ use session::context::QueryContext; use crate::hint_headers; -pub async fn extract_hints(mut request: Request, next: Next) -> Response { +pub async fn extract_hints(mut request: Request, next: Next) -> Response { let hints = hint_headers::extract_hints(request.headers()); if let Some(query_ctx) = request.extensions_mut().get_mut::() { for (key, value) in hints { diff --git a/src/servers/src/http/loki.rs b/src/servers/src/http/loki.rs index 272d2867db..05c2366a01 100644 --- a/src/servers/src/http/loki.rs +++ b/src/servers/src/http/loki.rs @@ -22,13 +22,14 @@ use api::v1::{ Value as GreptimeValue, }; use axum::extract::State; -use axum::headers::ContentType; -use axum::{Extension, TypedHeader}; +use axum::Extension; +use axum_extra::TypedHeader; use bytes::Bytes; use common_query::prelude::GREPTIME_TIMESTAMP; use common_query::{Output, OutputData}; use common_telemetry::{error, warn}; use hashbrown::HashMap; +use headers::ContentType; use lazy_static::lazy_static; use loki_api::prost_types::Timestamp; use prost::Message; diff --git a/src/servers/src/http/opentsdb.rs b/src/servers/src/http/opentsdb.rs index 17faba7c7d..87fa2cf75c 100644 --- a/src/servers/src/http/opentsdb.rs +++ b/src/servers/src/http/opentsdb.rs @@ -15,11 +15,11 @@ use std::collections::HashMap; use std::sync::Arc; -use axum::extract::{Query, RawBody, State}; +use axum::body::Bytes; +use axum::extract::{Query, State}; use axum::http::StatusCode as HttpStatusCode; use axum::{Extension, Json}; use common_error::ext::ErrorExt; -use hyper::Body; use serde::{Deserialize, Serialize}; use session::context::{Channel, QueryContext}; use snafu::ResultExt; @@ -76,7 +76,7 @@ pub async fn put( State(opentsdb_handler): State, Query(params): Query>, Extension(mut ctx): Extension, - RawBody(body): RawBody, + body: Bytes, ) -> Result<(HttpStatusCode, Json)> { let summary = params.contains_key("summary"); let details = params.contains_key("details"); @@ -125,10 +125,7 @@ pub async fn put( Ok(response) } -async fn parse_data_points(body: Body) -> Result> { - let body = hyper::body::to_bytes(body) - .await - .context(error::HyperSnafu)?; +async fn parse_data_points(body: Bytes) -> Result> { let data_points = serde_json::from_slice::>(&body[..]) .context(error::InvalidOpentsdbJsonRequestSnafu)?; Ok(data_points.into()) @@ -213,24 +210,24 @@ mod test { }"#; let data_point2 = serde_json::from_str::(raw_data_point2).unwrap(); - let body = Body::from(raw_data_point1); + let body = Bytes::from(raw_data_point1); let data_points = parse_data_points(body).await.unwrap(); assert_eq!(data_points.len(), 1); assert_eq!(data_points[0], data_point1); - let body = Body::from(format!("[{raw_data_point1},{raw_data_point2}]")); + let body = Bytes::from(format!("[{raw_data_point1},{raw_data_point2}]")); let data_points = parse_data_points(body).await.unwrap(); assert_eq!(data_points.len(), 2); assert_eq!(data_points[0], data_point1); assert_eq!(data_points[1], data_point2); - let body = Body::from(""); + let body = Bytes::from(""); let result = parse_data_points(body).await; assert!(result.is_err()); let err = result.unwrap_err().output_msg(); assert!(err.contains("EOF while parsing a value at line 1 column 0")); - let body = Body::from("hello world"); + let body = Bytes::from("hello world"); let result = parse_data_points(body).await; assert!(result.is_err()); let err = result.unwrap_err().output_msg(); diff --git a/src/servers/src/http/prom_store.rs b/src/servers/src/http/prom_store.rs index a429725819..581afcd57a 100644 --- a/src/servers/src/http/prom_store.rs +++ b/src/servers/src/http/prom_store.rs @@ -16,15 +16,16 @@ use std::sync::Arc; use api::prom_store::remote::ReadRequest; use api::v1::RowInsertRequests; -use axum::extract::{Query, RawBody, State}; +use axum::body::Bytes; +use axum::extract::{Query, State}; use axum::http::{header, HeaderValue, StatusCode}; use axum::response::IntoResponse; -use axum::{Extension, TypedHeader}; -use bytes::Bytes; +use axum::Extension; +use axum_extra::TypedHeader; use common_catalog::consts::DEFAULT_SCHEMA_NAME; use common_query::prelude::GREPTIME_PHYSICAL_TABLE; use common_telemetry::tracing; -use hyper::{Body, HeaderMap}; +use hyper::HeaderMap; use lazy_static::lazy_static; use object_pool::Pool; use prost::Message; @@ -75,7 +76,7 @@ pub async fn route_write_without_metric_engine( query: Query, extension: Extension, content_encoding: TypedHeader, - raw_body: RawBody, + raw_body: Bytes, ) -> Result { remote_write_impl( handler, @@ -97,7 +98,7 @@ pub async fn route_write_without_metric_engine_and_strict_mode( query: Query, extension: Extension, content_encoding: TypedHeader, - raw_body: RawBody, + raw_body: Bytes, ) -> Result { remote_write_impl( handler, @@ -121,7 +122,7 @@ pub async fn remote_write( query: Query, extension: Extension, content_encoding: TypedHeader, - raw_body: RawBody, + raw_body: Bytes, ) -> Result { remote_write_impl( handler, @@ -145,7 +146,7 @@ pub async fn remote_write_without_strict_mode( query: Query, extension: Extension, content_encoding: TypedHeader, - raw_body: RawBody, + raw_body: Bytes, ) -> Result { remote_write_impl( handler, @@ -164,7 +165,7 @@ async fn remote_write_impl( Query(params): Query, Extension(mut query_ctx): Extension, content_encoding: TypedHeader, - RawBody(body): RawBody, + body: Bytes, is_strict_mode: bool, is_metric_engine: bool, ) -> Result { @@ -224,7 +225,7 @@ pub async fn remote_read( State(handler): State, Query(params): Query, Extension(mut query_ctx): Extension, - RawBody(body): RawBody, + body: Bytes, ) -> Result { let db = params.db.clone().unwrap_or_default(); query_ctx.set_channel(Channel::Prometheus); @@ -248,13 +249,10 @@ fn try_decompress(is_zstd: bool, body: &[u8]) -> Result { async fn decode_remote_write_request( is_zstd: bool, - body: Body, + body: Bytes, is_strict_mode: bool, ) -> Result<(RowInsertRequests, usize)> { let _timer = crate::metrics::METRIC_HTTP_PROM_STORE_DECODE_ELAPSED.start_timer(); - let body = hyper::body::to_bytes(body) - .await - .context(error::HyperSnafu)?; // due to vmagent's limitation, there is a chance that vmagent is // sending content type wrong so we have to apply a fallback with decoding @@ -276,11 +274,7 @@ async fn decode_remote_write_request( Ok(request.as_row_insert_requests()) } -async fn decode_remote_read_request(body: Body) -> Result { - let body = hyper::body::to_bytes(body) - .await - .context(error::HyperSnafu)?; - +async fn decode_remote_read_request(body: Bytes) -> Result { let buf = snappy_decompress(&body[..])?; ReadRequest::decode(&buf[..]).context(error::DecodePromRemoteRequestSnafu) diff --git a/src/servers/src/http/result/greptime_manage_resp.rs b/src/servers/src/http/result/greptime_manage_resp.rs index 668a12f480..a46df298b4 100644 --- a/src/servers/src/http/result/greptime_manage_resp.rs +++ b/src/servers/src/http/result/greptime_manage_resp.rs @@ -97,10 +97,8 @@ impl IntoResponse for GreptimedbManageResponse { #[cfg(test)] mod tests { - use arrow::datatypes::ToByteSlice; - use http_body::Body; - use hyper::body::to_bytes; + use axum::body::to_bytes; use super::*; @@ -116,16 +114,14 @@ mod tests { execution_time_ms: 42, }; - let mut re = resp.into_response(); - let data = re.data(); - - let data_str = format!("{:?}", data); + let re = resp.into_response(); + let data_str = format!("{:?}", re); assert_eq!( data_str, - r#"Data(Response { status: 200, version: HTTP/1.1, headers: {"content-type": "application/json", "x-greptime-format": "greptimedb_manage", "x-greptime-execution-time": "42"}, body: UnsyncBoxBody })"# + r#"Response { status: 200, version: HTTP/1.1, headers: {"content-type": "application/json", "x-greptime-format": "greptimedb_manage", "x-greptime-execution-time": "42"}, body: Body(UnsyncBoxBody) }"# ); - let body_bytes = to_bytes(re.into_body()).await.unwrap(); + let body_bytes = to_bytes(re.into_body(), usize::MAX).await.unwrap(); let body_str = String::from_utf8_lossy(body_bytes.to_byte_slice()); assert_eq!( body_str, diff --git a/src/servers/src/http/result/greptime_result_v1.rs b/src/servers/src/http/result/greptime_result_v1.rs index 794308b1f9..ad19a13295 100644 --- a/src/servers/src/http/result/greptime_result_v1.rs +++ b/src/servers/src/http/result/greptime_result_v1.rs @@ -14,10 +14,10 @@ use std::collections::HashMap; -use axum::headers::HeaderValue; use axum::response::{IntoResponse, Response}; use axum::Json; use common_query::Output; +use headers::HeaderValue; use serde::{Deserialize, Serialize}; use serde_json::Value; diff --git a/src/servers/src/http/test_helpers.rs b/src/servers/src/http/test_helpers.rs index c5fe975a14..ca976220cf 100644 --- a/src/servers/src/http/test_helpers.rs +++ b/src/servers/src/http/test_helpers.rs @@ -13,7 +13,7 @@ //! let app = Router::new().route("/", get(|| async {})); //! //! // initiate the TestClient with the previous declared Router -//! let client = TestClient::new(app); +//! let client = TestClient::new(app).await; //! //! let res = client.get("/").await; //! assert_eq!(res.status(), StatusCode::OK); @@ -31,18 +31,14 @@ //! ``` use std::convert::TryFrom; -use std::net::{SocketAddr, TcpListener}; -use std::str::FromStr; +use std::net::SocketAddr; -use axum::body::HttpBody; -use axum::BoxError; +use axum::Router; use bytes::Bytes; use common_telemetry::info; use http::header::{HeaderName, HeaderValue}; -use http::{Request, StatusCode}; -use hyper::service::Service; -use hyper::{Body, Server}; -use tower::make::Shared; +use http::StatusCode; +use tokio::net::TcpListener; /// Test client to Axum servers. pub struct TestClient { @@ -52,22 +48,15 @@ pub struct TestClient { impl TestClient { /// Create a new test client. - pub fn new(svc: S) -> Self - where - S: Service, Response = http::Response> + Clone + Send + 'static, - ResBody: HttpBody + Send + 'static, - ResBody::Data: Send, - ResBody::Error: Into, - S::Future: Send, - S::Error: Into, - { - let listener = TcpListener::bind("127.0.0.1:0").expect("Could not bind ephemeral socket"); + pub async fn new(svc: Router) -> Self { + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("Could not bind ephemeral socket"); let addr = listener.local_addr().unwrap(); info!("Listening on {}", addr); tokio::spawn(async move { - let server = Server::from_tcp(listener).unwrap().serve(Shared::new(svc)); - server.await.expect("server error"); + axum::serve(listener, svc).await.expect("server error"); }); let client = reqwest::Client::builder() @@ -88,6 +77,8 @@ impl TestClient { /// Create a GET request. pub fn get(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("GET {} {}", self.addr, url); + RequestBuilder { builder: self.client.get(format!("http://{}{}", self.addr, url)), } @@ -95,6 +86,8 @@ impl TestClient { /// Create a HEAD request. pub fn head(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("HEAD {} {}", self.addr, url); + RequestBuilder { builder: self.client.head(format!("http://{}{}", self.addr, url)), } @@ -102,6 +95,8 @@ impl TestClient { /// Create a POST request. pub fn post(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("POST {} {}", self.addr, url); + RequestBuilder { builder: self.client.post(format!("http://{}{}", self.addr, url)), } @@ -109,6 +104,8 @@ impl TestClient { /// Create a PUT request. pub fn put(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("PUT {} {}", self.addr, url); + RequestBuilder { builder: self.client.put(format!("http://{}{}", self.addr, url)), } @@ -116,6 +113,8 @@ impl TestClient { /// Create a PATCH request. pub fn patch(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("PATCH {} {}", self.addr, url); + RequestBuilder { builder: self.client.patch(format!("http://{}{}", self.addr, url)), } @@ -123,6 +122,8 @@ impl TestClient { /// Create a DELETE request. pub fn delete(&self, url: &str) -> RequestBuilder { + common_telemetry::info!("DELETE {} {}", self.addr, url); + RequestBuilder { builder: self.client.delete(format!("http://{}{}", self.addr, url)), } @@ -170,13 +171,6 @@ impl RequestBuilder { HeaderValue: TryFrom, >::Error: Into, { - // TODO(tisonkun): revert once http bump to 1.x - let key: HeaderName = key.try_into().map_err(Into::into).unwrap(); - let key = reqwest::header::HeaderName::from_bytes(key.as_ref()).unwrap(); - - let value: HeaderValue = value.try_into().map_err(Into::into).unwrap(); - let value = reqwest::header::HeaderValue::from_bytes(value.as_bytes()).unwrap(); - self.builder = self.builder.header(key, value); self @@ -225,14 +219,7 @@ impl TestResponse { /// Get the response headers. pub fn headers(&self) -> http::HeaderMap { - // TODO(tisonkun): revert once http bump to 1.x - let mut headers = http::HeaderMap::new(); - for (key, value) in self.response.headers() { - let key = HeaderName::from_str(key.as_str()).unwrap(); - let value = HeaderValue::from_bytes(value.as_bytes()).unwrap(); - headers.insert(key, value); - } - headers + self.response.headers().clone() } /// Get the response in chunks. diff --git a/src/servers/src/http/timeout.rs b/src/servers/src/http/timeout.rs index c03620011e..050ec492e0 100644 --- a/src/servers/src/http/timeout.rs +++ b/src/servers/src/http/timeout.rs @@ -20,10 +20,10 @@ use std::time::Duration; use axum::body::Body; use axum::http::Request; use axum::response::Response; +use http::StatusCode; use pin_project::pin_project; use tokio::time::{Instant, Sleep}; -use tower::timeout::error::Elapsed; -use tower::{BoxError, Layer, Service}; +use tower::{Layer, Service}; use crate::http::header::constants::GREPTIME_DB_HEADER_TIMEOUT; @@ -31,49 +31,44 @@ use crate::http::header::constants::GREPTIME_DB_HEADER_TIMEOUT; /// /// [`Timeout`]: crate::timeout::Timeout /// -/// Modified from https://github.com/tower-rs/tower/blob/8b84b98d93a2493422a0ecddb6251f292a904cff/tower/src/timeout/future.rs +/// Modified from https://github.com/tower-rs/tower-http/blob/tower-http-0.5.2/tower-http/src/timeout/service.rs #[derive(Debug)] #[pin_project] pub struct ResponseFuture { #[pin] - response: T, + inner: T, #[pin] sleep: Sleep, } impl ResponseFuture { - pub(crate) fn new(response: T, sleep: Sleep) -> Self { - ResponseFuture { response, sleep } + pub(crate) fn new(inner: T, sleep: Sleep) -> Self { + ResponseFuture { inner, sleep } } } -impl Future for ResponseFuture +impl Future for ResponseFuture where - F: Future>, - E: Into, + F: Future>, { - type Output = Result; + type Output = Result; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { let this = self.project(); - // First, try polling the future - match this.response.poll(cx) { - Poll::Ready(v) => return Poll::Ready(v.map_err(Into::into)), - Poll::Pending => {} + if this.sleep.poll(cx).is_ready() { + let mut res = Response::default(); + *res.status_mut() = StatusCode::REQUEST_TIMEOUT; + return Poll::Ready(Ok(res)); } - // Now check the sleep - match this.sleep.poll(cx) { - Poll::Pending => Poll::Pending, - Poll::Ready(_) => Poll::Ready(Err(Elapsed::new().into())), - } + this.inner.poll(cx) } } /// Applies a timeout to requests via the supplied inner service. /// -/// Modified from https://github.com/tower-rs/tower/blob/8b84b98d93a2493422a0ecddb6251f292a904cff/tower/src/timeout/layer.rs +/// Modified from https://github.com/tower-rs/tower-http/blob/tower-http-0.5.2/tower-http/src/timeout/service.rs #[derive(Debug, Clone)] pub struct DynamicTimeoutLayer { default_timeout: Duration, @@ -94,7 +89,7 @@ impl Layer for DynamicTimeoutLayer { } } -/// Modified from https://github.com/tower-rs/tower/blob/8b84b98d93a2493422a0ecddb6251f292a904cff/tower/src/timeout/mod.rs +/// Modified from https://github.com/tower-rs/tower-http/blob/tower-http-0.5.2/tower-http/src/timeout/service.rs #[derive(Clone)] pub struct DynamicTimeout { inner: S, @@ -114,10 +109,9 @@ impl DynamicTimeout { impl Service> for DynamicTimeout where S: Service, Response = Response> + Send + 'static, - S::Error: Into, { type Response = S::Response; - type Error = BoxError; + type Error = S::Error; type Future = ResponseFuture; fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { diff --git a/src/servers/src/metrics.rs b/src/servers/src/metrics.rs index 2a3930e698..0489bb0d80 100644 --- a/src/servers/src/metrics.rs +++ b/src/servers/src/metrics.rs @@ -18,11 +18,9 @@ pub(crate) mod jemalloc; use std::task::{Context, Poll}; use std::time::Instant; -use axum::extract::MatchedPath; -use axum::http::Request; +use axum::extract::{MatchedPath, Request}; use axum::middleware::Next; use axum::response::IntoResponse; -use hyper::Body; use lazy_static::lazy_static; use prometheus::{ register_histogram_vec, register_int_counter, register_int_counter_vec, register_int_gauge, @@ -295,9 +293,9 @@ pub(crate) struct MetricsMiddleware { inner: S, } -impl Service> for MetricsMiddleware +impl Service> for MetricsMiddleware where - S: Service, Response = hyper::Response> + Clone + Send + 'static, + S: Service, Response = http::Response> + Clone + Send + 'static, S::Future: Send + 'static, { type Response = S::Response; @@ -308,7 +306,7 @@ where self.inner.poll_ready(cx) } - fn call(&mut self, req: hyper::Request) -> Self::Future { + fn call(&mut self, req: http::Request) -> Self::Future { // This is necessary because tonic internally uses `tower::buffer::Buffer`. // See https://github.com/tower-rs/tower/issues/547#issuecomment-767629149 // for details on why this is necessary @@ -338,7 +336,7 @@ where /// A middleware to record metrics for HTTP. // Based on https://github.com/tokio-rs/axum/blob/axum-v0.6.16/examples/prometheus-metrics/src/main.rs -pub(crate) async fn http_metrics_layer(req: Request, next: Next) -> impl IntoResponse { +pub(crate) async fn http_metrics_layer(req: Request, next: Next) -> impl IntoResponse { let start = Instant::now(); let path = if let Some(matched_path) = req.extensions().get::() { matched_path.as_str().to_owned() diff --git a/src/servers/src/mysql/helper.rs b/src/servers/src/mysql/helper.rs index c933fcc862..c5f8a1dd3e 100644 --- a/src/servers/src/mysql/helper.rs +++ b/src/servers/src/mysql/helper.rs @@ -328,11 +328,11 @@ mod tests { delete.inner.to_string() ); - let select = parse_sql("select from demo where host=? and idc in (select idc from idcs where name=?) and cpu>?"); + let select = parse_sql("select * from demo where host=? and idc in (select idc from idcs where name=?) and cpu>?"); let Statement::Query(select) = transform_placeholders(select) else { unreachable!() }; - assert_eq!("SELECT from AS demo WHERE host = $1 AND idc IN (SELECT idc FROM idcs WHERE name = $2) AND cpu > $3", select.inner.to_string()); + assert_eq!("SELECT * FROM demo WHERE host = $1 AND idc IN (SELECT idc FROM idcs WHERE name = $2) AND cpu > $3", select.inner.to_string()); } #[test] diff --git a/src/servers/src/postgres/types.rs b/src/servers/src/postgres/types.rs index 2a413ed00d..92a568caa2 100644 --- a/src/servers/src/postgres/types.rs +++ b/src/servers/src/postgres/types.rs @@ -938,7 +938,7 @@ pub(super) fn parameters_to_scalar_values( i.days, (i.microseconds / 1000) as i32, ) - .to_i64()) + .into()) } }) .transpose()?, @@ -946,7 +946,7 @@ pub(super) fn parameters_to_scalar_values( } ConcreteDataType::Interval(IntervalType::MonthDayNano(_)) => { ScalarValue::IntervalMonthDayNano( - data.map(|i| IntervalMonthDayNano::from(i).to_i128()), + data.map(|i| IntervalMonthDayNano::from(i).into()), ) } _ => { @@ -958,7 +958,7 @@ pub(super) fn parameters_to_scalar_values( } } else { ScalarValue::IntervalMonthDayNano( - data.map(|i| IntervalMonthDayNano::from(i).to_i128()), + data.map(|i| IntervalMonthDayNano::from(i).into()), ) } } @@ -1003,7 +1003,7 @@ pub(super) fn parameters_to_scalar_values( let data = portal.parameter::>(idx, &client_type)?; if let Some(data) = data { let values = data.into_iter().map(|i| i.into()).collect::>(); - ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int16)) + ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int16, true)) } else { ScalarValue::Null } @@ -1012,7 +1012,7 @@ pub(super) fn parameters_to_scalar_values( let data = portal.parameter::>(idx, &client_type)?; if let Some(data) = data { let values = data.into_iter().map(|i| i.into()).collect::>(); - ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int32)) + ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int32, true)) } else { ScalarValue::Null } @@ -1021,7 +1021,7 @@ pub(super) fn parameters_to_scalar_values( let data = portal.parameter::>(idx, &client_type)?; if let Some(data) = data { let values = data.into_iter().map(|i| i.into()).collect::>(); - ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int64)) + ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Int64, true)) } else { ScalarValue::Null } @@ -1030,7 +1030,7 @@ pub(super) fn parameters_to_scalar_values( let data = portal.parameter::>(idx, &client_type)?; if let Some(data) = data { let values = data.into_iter().map(|i| i.into()).collect::>(); - ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Utf8)) + ScalarValue::List(ScalarValue::new_list(&values, &ArrowDataType::Utf8, true)) } else { ScalarValue::Null } diff --git a/src/servers/tests/http/authorize.rs b/src/servers/tests/http/authorize.rs index fd825eed59..77da594b29 100644 --- a/src/servers/tests/http/authorize.rs +++ b/src/servers/tests/http/authorize.rs @@ -17,7 +17,6 @@ use std::sync::Arc; use auth::tests::MockUserProvider; use auth::UserProvider; use axum::http; -use http_body::Body; use hyper::{Request, StatusCode}; use servers::http::authorize::inner_auth; use servers::http::AUTHORIZATION_HEADER; @@ -46,11 +45,11 @@ async fn check_http_auth(header_key: &str) { let req = mock_http_request(header_key, None, None).unwrap(); let auth_res = inner_auth(mock_user_provider.clone(), req).await; assert!(auth_res.is_err()); - let mut resp = auth_res.unwrap_err(); + let resp = auth_res.unwrap_err(); assert_eq!(resp.status(), StatusCode::UNAUTHORIZED); assert_eq!( b"{\"code\":7003,\"error\":\"Not found http or grpc authorization header\",\"execution_time_ms\":0}", - resp.data().await.unwrap().unwrap().as_ref() + axum::body::to_bytes(resp.into_body(), usize::MAX).await.unwrap().as_ref() ); // base64encode("username:password") == "dXNlcm5hbWU6cGFzc3dvcmQ=" @@ -58,11 +57,14 @@ async fn check_http_auth(header_key: &str) { mock_http_request(header_key, Some("Basic dXNlcm5hbWU6cGFzc3dvcmQ="), None).unwrap(); let auth_res = inner_auth(mock_user_provider, wrong_req).await; assert!(auth_res.is_err()); - let mut resp = auth_res.unwrap_err(); + let resp = auth_res.unwrap_err(); assert_eq!(resp.status(), StatusCode::UNAUTHORIZED); assert_eq!( b"{\"code\":7000,\"error\":\"User not found, username: username\",\"execution_time_ms\":0}", - resp.data().await.unwrap().unwrap().as_ref(), + axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .unwrap() + .as_ref() ); } @@ -100,11 +102,11 @@ async fn check_schema_validating(header: &str) { .unwrap(); let result = inner_auth(mock_user_provider, req).await; assert!(result.is_err()); - let mut resp = result.unwrap_err(); + let resp = result.unwrap_err(); assert_eq!(resp.status(), StatusCode::UNAUTHORIZED); assert_eq!( b"{\"code\":7005,\"error\":\"Access denied for user 'greptime' to database 'greptime-wrong'\",\"execution_time_ms\":0}", - resp.data().await.unwrap().unwrap().as_ref() + axum::body::to_bytes(resp.into_body(), usize::MAX).await.unwrap().as_ref() ); } @@ -123,11 +125,11 @@ async fn check_auth_header(header_key: &str) { let req = mock_http_request(header_key, None, None).unwrap(); let auth_res = inner_auth(mock_user_provider.clone(), req).await; assert!(auth_res.is_err()); - let mut resp = auth_res.unwrap_err(); + let resp = auth_res.unwrap_err(); assert_eq!(resp.status(), StatusCode::UNAUTHORIZED); assert_eq!( b"{\"code\":7003,\"error\":\"Not found http or grpc authorization header\",\"execution_time_ms\":0}", - resp.data().await.unwrap().unwrap().as_ref() + axum::body::to_bytes(resp.into_body(), usize::MAX).await.unwrap().as_ref() ); // try whitelist path diff --git a/src/servers/tests/http/http_handler_test.rs b/src/servers/tests/http/http_handler_test.rs index 528e152e18..3f7f195bfd 100644 --- a/src/servers/tests/http/http_handler_test.rs +++ b/src/servers/tests/http/http_handler_test.rs @@ -16,12 +16,10 @@ use std::collections::HashMap; use axum::extract::{Json, Query, State}; use axum::http::header; -use axum::response::IntoResponse; +use axum::response::{IntoResponse, Response}; use axum::Form; use bytes::Bytes; use headers::HeaderValue; -use http_body::combinators::UnsyncBoxBody; -use hyper::Response; use mime_guess::mime; use servers::http::GreptimeQueryOutput::Records; use servers::http::{ @@ -94,7 +92,7 @@ async fn test_sql_output_rows() { "schema": { "column_schemas": [ { - "name": "SUM(numbers.uint32s)", + "name": "sum(numbers.uint32s)", "data_type": "UInt64" } ] @@ -121,7 +119,7 @@ async fn test_sql_output_rows() { { "name": "", "columns": [ - "SUM(numbers.uint32s)" + "sum(numbers.uint32s)" ], "values": [ [ @@ -135,28 +133,30 @@ async fn test_sql_output_rows() { ); } HttpResponse::Csv(resp) => { - use http_body::Body as HttpBody; - let mut resp = resp.into_response(); + let resp = resp.into_response(); assert_eq!( resp.headers().get(header::CONTENT_TYPE), Some(HeaderValue::from_static(mime::TEXT_CSV_UTF_8.as_ref())).as_ref(), ); assert_eq!( - resp.body_mut().data().await.unwrap().unwrap(), - hyper::body::Bytes::from_static(b"4950\n"), + axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .unwrap(), + Bytes::from_static(b"4950\n"), ); } HttpResponse::Table(resp) => { - use http_body::Body as HttpBody; - let mut resp = resp.into_response(); + let resp = resp.into_response(); assert_eq!( resp.headers().get(header::CONTENT_TYPE), Some(HeaderValue::from_static(mime::TEXT_PLAIN_UTF_8.as_ref())).as_ref(), ); assert_eq!( - resp.body_mut().data().await.unwrap().unwrap(), - hyper::body::Bytes::from( - r#"┌─SUM(numbers.uint32s)─┐ + axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .unwrap(), + Bytes::from( + r#"┌─sum(numbers.uint32s)─┐ │ 4950 │ └──────────────────────┘ "# @@ -239,7 +239,7 @@ async fn test_sql_form() { "schema": { "column_schemas": [ { - "name": "SUM(numbers.uint32s)", + "name": "sum(numbers.uint32s)", "data_type": "UInt64" } ] @@ -266,7 +266,7 @@ async fn test_sql_form() { { "name": "", "columns": [ - "SUM(numbers.uint32s)" + "sum(numbers.uint32s)" ], "values": [ [ @@ -280,28 +280,30 @@ async fn test_sql_form() { ); } HttpResponse::Csv(resp) => { - use http_body::Body as HttpBody; - let mut resp = resp.into_response(); + let resp = resp.into_response(); assert_eq!( resp.headers().get(header::CONTENT_TYPE), Some(HeaderValue::from_static(mime::TEXT_CSV_UTF_8.as_ref())).as_ref(), ); assert_eq!( - resp.body_mut().data().await.unwrap().unwrap(), - hyper::body::Bytes::from_static(b"4950\n"), + axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .unwrap(), + Bytes::from_static(b"4950\n"), ); } HttpResponse::Table(resp) => { - use http_body::Body as HttpBody; - let mut resp = resp.into_response(); + let resp = resp.into_response(); assert_eq!( resp.headers().get(header::CONTENT_TYPE), Some(HeaderValue::from_static(mime::TEXT_PLAIN_UTF_8.as_ref())).as_ref(), ); assert_eq!( - resp.body_mut().data().await.unwrap().unwrap(), - hyper::body::Bytes::from( - r#"┌─SUM(numbers.uint32s)─┐ + axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .unwrap(), + Bytes::from( + r#"┌─sum(numbers.uint32s)─┐ │ 4950 │ └──────────────────────┘ "# @@ -399,6 +401,8 @@ async fn test_config() { assert_eq!(get_body(rs).await, toml_str); } -async fn get_body(response: Response>) -> Bytes { - hyper::body::to_bytes(response.into_body()).await.unwrap() +async fn get_body(response: Response) -> Bytes { + axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap() } diff --git a/src/servers/tests/http/influxdb_test.rs b/src/servers/tests/http/influxdb_test.rs index 25b7d547af..b4f823da86 100644 --- a/src/servers/tests/http/influxdb_test.rs +++ b/src/servers/tests/http/influxdb_test.rs @@ -126,11 +126,13 @@ fn make_test_app(tx: Arc>, db_name: Option<&str>) #[tokio::test] async fn test_influxdb_write() { + common_telemetry::init_default_ut_logging(); + let (tx, mut rx) = mpsc::channel(100); let tx = Arc::new(tx); let app = make_test_app(tx.clone(), None); - let client = TestClient::new(app); + let client = TestClient::new(app).await; let result = client.get("/v1/influxdb/health").send().await; assert_eq!(result.status(), 200); @@ -197,7 +199,7 @@ async fn test_influxdb_write() { // make new app for db=influxdb let app = make_test_app(tx, Some("influxdb")); - let client = TestClient::new(app); + let client = TestClient::new(app).await; // right request let result = client @@ -236,11 +238,13 @@ async fn test_influxdb_write() { #[tokio::test] async fn test_influxdb_write_v2() { + common_telemetry::init_default_ut_logging(); + let (tx, mut rx) = mpsc::channel(100); let tx = Arc::new(tx); let public_db_app = make_test_app(tx.clone(), None); - let public_db_client = TestClient::new(public_db_app); + let public_db_client = TestClient::new(public_db_app).await; let result = public_db_client.get("/v1/influxdb/health").send().await; assert_eq!(result.status(), 200); @@ -280,7 +284,7 @@ async fn test_influxdb_write_v2() { // make new app for 'influxdb' database let app = make_test_app(tx, Some("influxdb")); - let client = TestClient::new(app); + let client = TestClient::new(app).await; // right request with `bucket` query string let result = client diff --git a/src/servers/tests/http/opentsdb_test.rs b/src/servers/tests/http/opentsdb_test.rs index c042c99b67..efbfc2a310 100644 --- a/src/servers/tests/http/opentsdb_test.rs +++ b/src/servers/tests/http/opentsdb_test.rs @@ -117,10 +117,12 @@ fn make_test_app(tx: mpsc::Sender) -> Router { #[tokio::test] async fn test_opentsdb_put() { + common_telemetry::init_default_ut_logging(); + let (tx, mut rx) = mpsc::channel(100); let app = make_test_app(tx); - let client = TestClient::new(app); + let client = TestClient::new(app).await; // single data point put let result = client @@ -174,10 +176,12 @@ async fn test_opentsdb_put() { #[tokio::test] async fn test_opentsdb_debug_put() { + common_telemetry::init_default_ut_logging(); + let (tx, mut rx) = mpsc::channel(100); let app = make_test_app(tx); - let client = TestClient::new(app); + let client = TestClient::new(app).await; // single data point summary debug put let result = client diff --git a/src/servers/tests/http/prom_store_test.rs b/src/servers/tests/http/prom_store_test.rs index 8e70007cbd..f5ca54d22d 100644 --- a/src/servers/tests/http/prom_store_test.rs +++ b/src/servers/tests/http/prom_store_test.rs @@ -150,7 +150,7 @@ async fn test_prometheus_remote_write_read() { let (tx, mut rx) = mpsc::channel(100); let app = make_test_app(tx); - let client = TestClient::new(app); + let client = TestClient::new(app).await; let write_request = WriteRequest { timeseries: prom_store::mock_timeseries(), diff --git a/src/sql/src/ast.rs b/src/sql/src/ast.rs index e1944de98a..519f13154c 100644 --- a/src/sql/src/ast.rs +++ b/src/sql/src/ast.rs @@ -14,6 +14,7 @@ pub use sqlparser::ast::{ visit_expressions_mut, visit_statements_mut, BinaryOperator, ColumnDef, ColumnOption, - ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName, - SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor, VisitorMut, + ColumnOptionDef, DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArguments, + Ident, ObjectName, SqlOption, TableConstraint, TimezoneInfo, Value, Visit, VisitMut, Visitor, + VisitorMut, }; diff --git a/src/sql/src/error.rs b/src/sql/src/error.rs index 25cb6578de..e7253d6c46 100644 --- a/src/sql/src/error.rs +++ b/src/sql/src/error.rs @@ -72,7 +72,7 @@ pub enum Error { }, // Syntax error from sql parser. - #[snafu(display(""))] + #[snafu(display("Invalid SQL syntax"))] Syntax { #[snafu(source)] error: ParserError, @@ -81,7 +81,7 @@ pub enum Error { }, // Syntax error from tql parser. - #[snafu(display(""))] + #[snafu(display("Invalid TQL syntax"))] TQLSyntax { #[snafu(source)] error: TQLError, diff --git a/src/sql/src/lib.rs b/src/sql/src/lib.rs index 33dfdc215d..8f2ca7fc4f 100644 --- a/src/sql/src/lib.rs +++ b/src/sql/src/lib.rs @@ -14,6 +14,7 @@ #![feature(box_patterns)] #![feature(assert_matches)] +#![feature(if_let_guard)] #![feature(let_chains)] pub mod ast; diff --git a/src/sql/src/parser.rs b/src/sql/src/parser.rs index 91f6a14040..087e5a915c 100644 --- a/src/sql/src/parser.rs +++ b/src/sql/src/parser.rs @@ -50,7 +50,7 @@ impl ParserContext<'_> { /// Parses parser context to Query. pub fn parser_query(&mut self) -> Result> { - Ok(Box::new(self.parser.parse_query().context(SyntaxSnafu)?)) + self.parser.parse_query().context(SyntaxSnafu) } /// Parses SQL with given dialect diff --git a/src/sql/src/parsers/admin_parser.rs b/src/sql/src/parsers/admin_parser.rs index a9644bc39d..13f9f9f833 100644 --- a/src/sql/src/parsers/admin_parser.rs +++ b/src/sql/src/parsers/admin_parser.rs @@ -46,6 +46,8 @@ impl ParserContext<'_> { #[cfg(test)] mod tests { + use sqlparser::ast::FunctionArguments; + use super::*; use crate::ast::{Expr, Function, FunctionArg, FunctionArgExpr, Value}; use crate::dialect::GreptimeDbDialect; @@ -62,9 +64,12 @@ mod tests { let stmt = result.remove(0); match &stmt { Statement::Admin(Admin::Func(Function { name, args, .. })) => { + let FunctionArguments::List(arg_list) = args else { + unreachable!() + }; assert_eq!("flush_table", name.to_string()); - assert_eq!(args.len(), 1); - assert!(matches!(&args[0], + assert_eq!(arg_list.args.len(), 1); + assert!(matches!(&arg_list.args[0], FunctionArg::Unnamed(FunctionArgExpr::Expr( Expr::Value(Value::SingleQuotedString(s)) )) if s == "test")); @@ -86,8 +91,11 @@ mod tests { let stmt = result.remove(0); match &stmt { Statement::Admin(Admin::Func(Function { name, args, .. })) => { + let FunctionArguments::List(arg_list) = args else { + unreachable!() + }; assert_eq!("test", name.to_string()); - assert_eq!(args.len(), 0); + assert_eq!(arg_list.args.len(), 0); } _ => unreachable!(), } diff --git a/src/sql/src/parsers/alter_parser.rs b/src/sql/src/parsers/alter_parser.rs index cf7015607d..87636efed6 100644 --- a/src/sql/src/parsers/alter_parser.rs +++ b/src/sql/src/parsers/alter_parser.rs @@ -644,7 +644,7 @@ mod tests { let err = result.output_msg(); assert_eq!( err, - "sql parser error: Expected COLUMN, found: a at Line: 1, Column 30" + "Invalid SQL syntax: sql parser error: Expected: COLUMN, found: a at Line: 1, Column: 30" ); let sql = "ALTER TABLE my_metric_1 DROP COLUMN a"; @@ -790,7 +790,7 @@ mod tests { ParserContext::create_with_dialect(sql, &GreptimeDbDialect {}, ParseOptions::default()) .unwrap_err(); let err = result.output_msg(); - assert_eq!(err, "sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t"); + assert_eq!(err, "Invalid SQL syntax: sql parser error: Expected ADD or DROP or MODIFY or RENAME or SET after ALTER TABLE, found: table_t"); let sql = "ALTER TABLE test_table RENAME table_t"; let mut result = diff --git a/src/sql/src/parsers/create_parser.rs b/src/sql/src/parsers/create_parser.rs index f40ecb7b6e..f371788a47 100644 --- a/src/sql/src/parsers/create_parser.rs +++ b/src/sql/src/parsers/create_parser.rs @@ -44,9 +44,8 @@ use crate::statements::create::{ CreateTableLike, CreateView, Partitions, TableConstraint, VECTOR_OPT_DIM, }; use crate::statements::statement::Statement; -use crate::statements::{ - get_data_type_by_alias_name, sql_data_type_to_concrete_data_type, OptionMap, -}; +use crate::statements::transform::type_alias::get_data_type_by_alias_name; +use crate::statements::{sql_data_type_to_concrete_data_type, OptionMap}; use crate::util::parse_option_string; pub const ENGINE: &str = "ENGINE"; @@ -291,13 +290,7 @@ impl<'a> ParserContext<'a> { reason: format!("cannot cast {} to interval type", expire_after_expr), })?; if let ScalarValue::IntervalMonthDayNano(Some(nanoseconds)) = expire_after_lit { - Some( - i64::try_from(nanoseconds / 1_000_000_000) - .ok() - .with_context(|| InvalidIntervalSnafu { - reason: format!("interval {} overflows", nanoseconds), - })?, - ) + Some(nanoseconds.nanoseconds / 1_000_000_000) } else { unreachable!() } @@ -326,7 +319,7 @@ impl<'a> ParserContext<'a> { .expect_keyword(Keyword::AS) .context(SyntaxSnafu)?; - let query = Box::new(self.parser.parse_query().context(error::SyntaxSnafu)?); + let query = self.parser.parse_query().context(error::SyntaxSnafu)?; Ok(Statement::CreateFlow(CreateFlow { flow_name, @@ -1179,7 +1172,7 @@ mod tests { assert_column_def(&columns[0].column_def, "host", "STRING"); assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP"); assert_column_def(&columns[2].column_def, "cpu", "FLOAT"); - assert_column_def(&columns[3].column_def, "memory", "FLOAT64"); + assert_column_def(&columns[3].column_def, "memory", "DOUBLE"); let constraints = &c.constraints; assert_eq!( @@ -1758,7 +1751,7 @@ ENGINE=mito"; assert!(result .unwrap_err() .output_msg() - .contains("sql parser error: Expected ON, found: COLUMNS")); + .contains("sql parser error: Expected: ON, found: COLUMNS")); } #[test] @@ -1835,7 +1828,7 @@ ENGINE=mito"; assert_column_def(&columns[0].column_def, "host", "STRING"); assert_column_def(&columns[1].column_def, "ts", "TIMESTAMP"); assert_column_def(&columns[2].column_def, "cpu", "FLOAT"); - assert_column_def(&columns[3].column_def, "memory", "FLOAT64"); + assert_column_def(&columns[3].column_def, "memory", "DOUBLE"); let constraints = &c.constraints; assert_eq!( diff --git a/src/sql/src/parsers/explain_parser.rs b/src/sql/src/parsers/explain_parser.rs index 719a0663cc..7574c4a68f 100644 --- a/src/sql/src/parsers/explain_parser.rs +++ b/src/sql/src/parsers/explain_parser.rs @@ -68,12 +68,13 @@ mod tests { with_hints: vec![], partitions: vec![], version: None, + with_ordinality: false, }, joins: vec![], }], lateral_views: vec![], selection: None, - group_by: GroupByExpr::Expressions(vec![]), + group_by: GroupByExpr::Expressions(vec![], vec![]), cluster_by: vec![], distribute_by: vec![], sort_by: vec![], @@ -81,18 +82,24 @@ mod tests { qualify: None, named_window: vec![], value_table_mode: None, + top_before_distinct: false, + prewhere: None, + window_before_qualify: false, + connect_by: None, }; let sp_statement = SpStatement::Query(Box::new(SpQuery { with: None, body: Box::new(sqlparser::ast::SetExpr::Select(Box::new(select))), - order_by: vec![], + order_by: None, limit: None, limit_by: vec![], offset: None, fetch: None, locks: vec![], for_clause: None, + settings: None, + format_clause: None, })); let explain = Explain::try_from(SpStatement::Explain { @@ -101,6 +108,8 @@ mod tests { verbose: false, statement: Box::new(sp_statement), format: None, + query_plan: false, + options: None, }) .unwrap(); diff --git a/src/sql/src/parsers/query_parser.rs b/src/sql/src/parsers/query_parser.rs index f2fbea1b3c..fc3cecf1a3 100644 --- a/src/sql/src/parsers/query_parser.rs +++ b/src/sql/src/parsers/query_parser.rs @@ -24,7 +24,7 @@ impl ParserContext<'_> { pub(crate) fn parse_query(&mut self) -> Result { let spquery = self.parser.parse_query().context(error::SyntaxSnafu)?; - Ok(Statement::Query(Box::new(Query::try_from(spquery)?))) + Ok(Statement::Query(Box::new(Query::try_from(*spquery)?))) } } @@ -56,6 +56,6 @@ mod tests { assert!(result .unwrap_err() .output_msg() - .contains("Expected an expression")); + .contains("Expected: an expression")); } } diff --git a/src/sql/src/parsers/set_var_parser.rs b/src/sql/src/parsers/set_var_parser.rs index 8a66269803..63119198ae 100644 --- a/src/sql/src/parsers/set_var_parser.rs +++ b/src/sql/src/parsers/set_var_parser.rs @@ -28,11 +28,14 @@ impl ParserContext<'_> { let spstatement = self.parser.parse_set().context(error::SyntaxSnafu)?; match spstatement { SpStatement::SetVariable { - variable, + variables, value, hivevar, .. - } if !hivevar => Ok(Statement::SetVariables(SetVariables { variable, value })), + } if !hivevar => Ok(Statement::SetVariables(SetVariables { + variable: (*variables)[0].clone(), + value, + })), SpStatement::SetTimeZone { value, .. } => Ok(Statement::SetVariables(SetVariables { variable: ObjectName(vec![Ident { diff --git a/src/sql/src/parsers/utils.rs b/src/sql/src/parsers/utils.rs index f7eefc4b95..ef6a3283eb 100644 --- a/src/sql/src/parsers/utils.rs +++ b/src/sql/src/parsers/utils.rs @@ -18,6 +18,7 @@ use chrono::Utc; use datafusion::config::ConfigOptions; use datafusion::error::Result as DfResult; use datafusion::execution::context::SessionState; +use datafusion::execution::SessionStateBuilder; use datafusion::optimizer::simplify_expressions::ExprSimplifier; use datafusion_common::{DFSchema, ScalarValue}; use datafusion_expr::execution_props::ExecutionProps; @@ -72,7 +73,11 @@ struct StubContextProvider { impl Default for StubContextProvider { fn default() -> Self { Self { - state: SessionState::new_with_config_rt(Default::default(), Default::default()), + state: SessionStateBuilder::new() + .with_config(Default::default()) + .with_runtime_env(Default::default()) + .with_default_features() + .build(), } } } diff --git a/src/sql/src/statements.rs b/src/sql/src/statements.rs index ec6fec0304..21e9ead437 100644 --- a/src/sql/src/statements.rs +++ b/src/sql/src/statements.rs @@ -28,7 +28,7 @@ pub mod set_variables; pub mod show; pub mod statement; pub mod tql; -mod transform; +pub(crate) mod transform; pub mod truncate; use std::str::FromStr; @@ -61,7 +61,7 @@ use crate::error::{ }; use crate::statements::create::Column; pub use crate::statements::option_map::OptionMap; -pub use crate::statements::transform::{get_data_type_by_alias_name, transform_statements}; +pub(crate) use crate::statements::transform::transform_statements; const VECTOR_TYPE_NAME: &str = "VECTOR"; diff --git a/src/sql/src/statements/create.rs b/src/sql/src/statements/create.rs index 3ea265fb7f..6fd0333a80 100644 --- a/src/sql/src/statements/create.rs +++ b/src/sql/src/statements/create.rs @@ -695,8 +695,8 @@ CREATE TABLE t2 LIKE t1"#, CREATE EXTERNAL TABLE city ( host STRING, ts TIMESTAMP, - cpu FLOAT64 DEFAULT 0, - memory FLOAT64, + cpu DOUBLE DEFAULT 0, + memory DOUBLE, TIME INDEX (ts), PRIMARY KEY (host) ) diff --git a/src/sql/src/statements/insert.rs b/src/sql/src/statements/insert.rs index f1c0b71444..6dad03fe95 100644 --- a/src/sql/src/statements/insert.rs +++ b/src/sql/src/statements/insert.rs @@ -13,7 +13,9 @@ // limitations under the License. use serde::Serialize; -use sqlparser::ast::{ObjectName, Query, SetExpr, Statement, UnaryOperator, Values}; +use sqlparser::ast::{ + Insert as SpInsert, ObjectName, Query, SetExpr, Statement, UnaryOperator, Values, +}; use sqlparser::parser::ParserError; use sqlparser_derive::{Visit, VisitMut}; @@ -39,14 +41,14 @@ macro_rules! parse_fail { impl Insert { pub fn table_name(&self) -> &ObjectName { match &self.inner { - Statement::Insert { table_name, .. } => table_name, + Statement::Insert(insert) => &insert.table_name, _ => unreachable!(), } } pub fn columns(&self) -> Vec<&String> { match &self.inner { - Statement::Insert { columns, .. } => columns.iter().map(|ident| &ident.value).collect(), + Statement::Insert(insert) => insert.columns.iter().map(|ident| &ident.value).collect(), _ => unreachable!(), } } @@ -54,14 +56,14 @@ impl Insert { /// Extracts the literal insert statement body if possible pub fn values_body(&self) -> Result>> { match &self.inner { - Statement::Insert { + Statement::Insert(SpInsert { source: Some(box Query { body: box SetExpr::Values(Values { rows, .. }), .. }), .. - } => sql_exprs_to_values(rows), + }) => sql_exprs_to_values(rows), _ => unreachable!(), } } @@ -70,14 +72,14 @@ impl Insert { /// The rules is the same as function `values_body()`. pub fn can_extract_values(&self) -> bool { match &self.inner { - Statement::Insert { + Statement::Insert(SpInsert { source: Some(box Query { body: box SetExpr::Values(Values { rows, .. }), .. }), .. - } => rows.iter().all(|es| { + }) => rows.iter().all(|es| { es.iter().all(|expr| match expr { Expr::Value(_) => true, Expr::Identifier(ident) => { @@ -100,10 +102,10 @@ impl Insert { pub fn query_body(&self) -> Result> { Ok(match &self.inner { - Statement::Insert { + Statement::Insert(SpInsert { source: Some(box query), .. - } => Some(query.clone().try_into()?), + }) => Some(query.clone().try_into()?), _ => None, }) } diff --git a/src/sql/src/statements/transform.rs b/src/sql/src/statements/transform.rs index 7f3c523cfc..2ca642cd11 100644 --- a/src/sql/src/statements/transform.rs +++ b/src/sql/src/statements/transform.rs @@ -12,20 +12,19 @@ // See the License for the specific language governing permissions and // limitations under the License. +mod expand_interval; +pub(crate) mod type_alias; + use std::ops::ControlFlow; use std::sync::Arc; +use expand_interval::ExpandIntervalTransformRule; use lazy_static::lazy_static; use sqlparser::ast::{visit_expressions_mut, Expr}; +use type_alias::TypeAliasTransformRule; use crate::error::Result; use crate::statements::statement::Statement; -mod expand_interval; -mod type_alias; - -use expand_interval::ExpandIntervalTransformRule; -pub use type_alias::get_data_type_by_alias_name; -use type_alias::TypeAliasTransformRule; lazy_static! { /// [TransformRule] registry diff --git a/src/sql/src/statements/transform/expand_interval.rs b/src/sql/src/statements/transform/expand_interval.rs index 084a938c8f..9761dcb0d7 100644 --- a/src/sql/src/statements/transform/expand_interval.rs +++ b/src/sql/src/statements/transform/expand_interval.rs @@ -371,6 +371,7 @@ mod tests { expr: single_quoted_string_expr("3y2mon".to_string()), data_type: DataType::Interval, format: None, + kind: sqlparser::ast::CastKind::Cast, }; let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_interval_expr); @@ -393,6 +394,7 @@ mod tests { expr: single_quoted_string_expr("5".to_string()), data_type: DataType::Int64, format: None, + kind: sqlparser::ast::CastKind::Cast, }; let control_flow = interval_transformation_rule.visit_expr(&mut cast_to_i64_expr); assert_eq!(control_flow, ControlFlow::Continue(())); @@ -402,6 +404,7 @@ mod tests { expr: single_quoted_string_expr("5".to_string()), data_type: DataType::Int64, format: None, + kind: sqlparser::ast::CastKind::Cast, } ); } diff --git a/src/sql/src/statements/transform/type_alias.rs b/src/sql/src/statements/transform/type_alias.rs index 4d2f73acbd..bca3160fe3 100644 --- a/src/sql/src/statements/transform/type_alias.rs +++ b/src/sql/src/statements/transform/type_alias.rs @@ -16,7 +16,8 @@ use std::ops::ControlFlow; use datatypes::data_type::DataType as GreptimeDataType; use sqlparser::ast::{ - DataType, Expr, Function, FunctionArg, FunctionArgExpr, Ident, ObjectName, Value, + DataType, Expr, Function, FunctionArg, FunctionArgExpr, FunctionArgumentList, Ident, + ObjectName, Value, }; use crate::error::Result; @@ -75,18 +76,21 @@ impl TransformRule for TypeAliasTransformRule { fn cast_expr_to_arrow_cast_func(expr: Expr, cast_type: String) -> Function { Function { name: ObjectName(vec![Ident::new("arrow_cast")]), - args: vec![ - FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)), - FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value( - Value::SingleQuotedString(cast_type), - ))), - ], + args: sqlparser::ast::FunctionArguments::List(FunctionArgumentList { + args: vec![ + FunctionArg::Unnamed(FunctionArgExpr::Expr(expr)), + FunctionArg::Unnamed(FunctionArgExpr::Expr(Expr::Value( + Value::SingleQuotedString(cast_type), + ))), + ], + duplicate_treatment: None, + clauses: vec![], + }), filter: None, null_treatment: None, over: None, - distinct: false, - special: false, - order_by: vec![], + parameters: sqlparser::ast::FunctionArguments::None, + within_group: vec![], } } @@ -99,30 +103,14 @@ impl TransformRule for TypeAliasTransformRule { expr: cast_expr, data_type, .. - } if matches!(data_type, DataType::Int64 | DataType::Float64) => { - if let Some(new_type) = get_data_type_by_alias_name(&data_type.to_string()) { - if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) { - *expr = Expr::Function(cast_expr_to_arrow_cast_func( - (**cast_expr).clone(), - new_type.as_arrow_type().to_string(), - )); - } - } - } - - // Type alias - Expr::Cast { - data_type: DataType::Custom(name, tokens), - expr: cast_expr, - .. - } if name.0.len() == 1 && tokens.is_empty() => { - if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) { - if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) { - *expr = Expr::Function(cast_expr_to_arrow_cast_func( - (**cast_expr).clone(), - new_type.as_arrow_type().to_string(), - )); - } + } if get_type_by_alias(data_type).is_some() => { + // Safety: checked in the match arm. + let new_type = get_type_by_alias(data_type).unwrap(); + if let Ok(new_type) = sql_data_type_to_concrete_data_type(&new_type) { + *expr = Expr::Function(cast_expr_to_arrow_cast_func( + (**cast_expr).clone(), + new_type.as_arrow_type().to_string(), + )); } } @@ -153,20 +141,44 @@ impl TransformRule for TypeAliasTransformRule { } fn replace_type_alias(data_type: &mut DataType) { - match data_type { - // TODO(dennis): The sqlparser latest version contains the Int8 alias for Postgres Bigint. - // Which means 8 bytes in postgres (not 8 bits). If we upgrade the sqlparser, need to process it. - // See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8 - DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => { - if let Some(new_type) = get_data_type_by_alias_name(name.0[0].value.as_str()) { - *data_type = new_type; - } - } - _ => {} + if let Some(new_type) = get_type_by_alias(data_type) { + *data_type = new_type; } } -pub fn get_data_type_by_alias_name(name: &str) -> Option { +/// Get data type from alias type. +/// Returns the mapped data type if the input data type is an alias that we need to replace. +// Remember to update `get_data_type_by_alias_name()` if you modify this method. +pub(crate) fn get_type_by_alias(data_type: &DataType) -> Option { + match data_type { + // The sqlparser latest version contains the Int8 alias for Postgres Bigint. + // Which means 8 bytes in postgres (not 8 bits). + // See https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Int8 + DataType::Custom(name, tokens) if name.0.len() == 1 && tokens.is_empty() => { + get_data_type_by_alias_name(name.0[0].value.as_str()) + } + DataType::Int8(None) => Some(DataType::TinyInt(None)), + DataType::Int16 => Some(DataType::SmallInt(None)), + DataType::Int32 => Some(DataType::Int(None)), + DataType::Int64 => Some(DataType::BigInt(None)), + DataType::UInt8 => Some(DataType::UnsignedTinyInt(None)), + DataType::UInt16 => Some(DataType::UnsignedSmallInt(None)), + DataType::UInt32 => Some(DataType::UnsignedInt(None)), + DataType::UInt64 => Some(DataType::UnsignedBigInt(None)), + DataType::Float32 => Some(DataType::Float(None)), + DataType::Float64 => Some(DataType::Double), + _ => None, + } +} + +/// Get the mapped data type from alias name. +/// It only supports the following types of alias: +/// - timestamps +/// - ints +/// - floats +/// - texts +// Remember to update `get_type_alias()` if you modify this method. +pub(crate) fn get_data_type_by_alias_name(name: &str) -> Option { match name.to_uppercase().as_ref() { // Timestamp type alias "TIMESTAMP_S" | "TIMESTAMP_SEC" | "TIMESTAMPSECOND" => { @@ -183,6 +195,7 @@ pub fn get_data_type_by_alias_name(name: &str) -> Option { Some(DataType::Timestamp(Some(9), TimezoneInfo::None)) } // Number type alias + // We keep them for backward compatibility. "INT8" => Some(DataType::TinyInt(None)), "INT16" => Some(DataType::SmallInt(None)), "INT32" => Some(DataType::Int(None)), @@ -384,13 +397,13 @@ CREATE TABLE data_types ( tt TEXT, mt TEXT, lt TEXT, - tint INT8, + tint TINYINT, sint SMALLINT, i INT, - bint INT64, + bint BIGINT, v VARCHAR, f FLOAT, - d FLOAT64, + d DOUBLE, b BOOLEAN, vb VARBINARY, dt DATE, diff --git a/src/sql/src/util.rs b/src/sql/src/util.rs index 7d8cf3df50..92eb8fedbe 100644 --- a/src/sql/src/util.rs +++ b/src/sql/src/util.rs @@ -17,7 +17,7 @@ use std::fmt::{Display, Formatter}; use sqlparser::ast::{Expr, ObjectName, Query, SetExpr, SqlOption, TableFactor, Value}; -use crate::error::{InvalidTableOptionValueSnafu, Result}; +use crate::error::{InvalidSqlSnafu, InvalidTableOptionValueSnafu, Result}; /// Format an [ObjectName] without any quote of its idents. pub fn format_raw_object_name(name: &ObjectName) -> String { @@ -41,7 +41,12 @@ pub fn format_raw_object_name(name: &ObjectName) -> String { } pub fn parse_option_string(option: SqlOption) -> Result<(String, String)> { - let (key, value) = (option.name, option.value); + let SqlOption::KeyValue { key, value } = option else { + return InvalidSqlSnafu { + msg: "Expecting a key-value pair in the option", + } + .fail(); + }; let v = match value { Expr::Value(Value::SingleQuotedString(v)) | Expr::Value(Value::DoubleQuotedString(v)) => v, Expr::Identifier(v) => v.value, diff --git a/src/store-api/src/metadata.rs b/src/store-api/src/metadata.rs index 4706aa65cd..febb0bc28f 100644 --- a/src/store-api/src/metadata.rs +++ b/src/store-api/src/metadata.rs @@ -240,6 +240,11 @@ impl RegionMetadata { &self.column_metadatas[index] } + /// Returns the position of the time index. + pub fn time_index_column_pos(&self) -> usize { + self.id_to_index[&self.time_index] + } + /// Returns the arrow field of the time index column. pub fn time_index_field(&self) -> FieldRef { let index = self.id_to_index[&self.time_index]; @@ -918,7 +923,7 @@ pub enum MetadataError { #[snafu(display("Failed to decode protobuf"))] DecodeProto { #[snafu(source)] - error: prost::DecodeError, + error: prost::UnknownEnumValue, #[snafu(implicit)] location: Location, }, diff --git a/src/table/src/predicate/stats.rs b/src/table/src/predicate/stats.rs index 03f84a110b..38c01648e8 100644 --- a/src/table/src/predicate/stats.rs +++ b/src/table/src/predicate/stats.rs @@ -64,25 +64,21 @@ macro_rules! impl_min_max_values { .iter() .map(|meta| { let stats = meta.column(column_index).statistics()?; - if !stats.has_min_max_set() { - return None; - } - match stats { - ParquetStats::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.$min_max()))), - ParquetStats::Int32(s) => Some(ScalarValue::Int32(Some(*s.$min_max()))), - ParquetStats::Int64(s) => Some(ScalarValue::Int64(Some(*s.$min_max()))), + paste! { + match stats { + ParquetStats::Boolean(s) => Some(ScalarValue::Boolean(Some(*s.[<$min_max _opt>]()?))), + ParquetStats::Int32(s) => Some(ScalarValue::Int32(Some(*s.[<$min_max _opt>]()?))), + ParquetStats::Int64(s) => Some(ScalarValue::Int64(Some(*s.[<$min_max _opt>]()?))), - ParquetStats::Int96(_) => None, - ParquetStats::Float(s) => Some(ScalarValue::Float32(Some(*s.$min_max()))), - ParquetStats::Double(s) => Some(ScalarValue::Float64(Some(*s.$min_max()))), - ParquetStats::ByteArray(s) => { - paste! { - let s = String::from_utf8(s.[<$min_max _bytes>]().to_owned()).ok(); + ParquetStats::Int96(_) => None, + ParquetStats::Float(s) => Some(ScalarValue::Float32(Some(*s.[<$min_max _opt>]()?))), + ParquetStats::Double(s) => Some(ScalarValue::Float64(Some(*s.[<$min_max _opt>]()?))), + ParquetStats::ByteArray(s) => { + let s = String::from_utf8(s.[<$min_max _bytes_opt>]()?.to_owned()).ok(); + Some(ScalarValue::Utf8(s)) } - Some(ScalarValue::Utf8(s)) + ParquetStats::FixedLenByteArray(_) => None, } - - ParquetStats::FixedLenByteArray(_) => None, } }) .map(|maybe_scalar| maybe_scalar.unwrap_or_else(|| null_scalar.clone())) @@ -111,7 +107,7 @@ impl PruningStatistics for RowGroupPruningStatistics<'_> { for m in self.meta_data { let col = m.column(idx); let stat = col.statistics()?; - let bs = stat.null_count(); + let bs = stat.null_count_opt()?; values.push(Some(bs)); } Some(Arc::new(UInt64Array::from(values))) diff --git a/src/table/src/table/adapter.rs b/src/table/src/table/adapter.rs index f191450a75..2cf9a9647f 100644 --- a/src/table/src/table/adapter.rs +++ b/src/table/src/table/adapter.rs @@ -18,9 +18,9 @@ use std::sync::{Arc, Mutex}; use common_query::stream::StreamScanAdapter; use common_recordbatch::OrderOption; use datafusion::arrow::datatypes::SchemaRef as DfSchemaRef; +use datafusion::catalog::Session; use datafusion::datasource::{TableProvider, TableType as DfTableType}; use datafusion::error::Result as DfResult; -use datafusion::execution::context::SessionState; use datafusion::physical_plan::ExecutionPlan; use datafusion_expr::expr::Expr; use datafusion_expr::TableProviderFilterPushDown as DfTableProviderFilterPushDown; @@ -58,6 +58,14 @@ impl DfTableProviderAdapter { } } +impl std::fmt::Debug for DfTableProviderAdapter { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("DfTableProviderAdapter") + .field("table", &self.table.table_info.full_table_name()) + .finish() + } +} + #[async_trait::async_trait] impl TableProvider for DfTableProviderAdapter { fn as_any(&self) -> &dyn Any { @@ -82,7 +90,7 @@ impl TableProvider for DfTableProviderAdapter { async fn scan( &self, - _ctx: &SessionState, + _state: &dyn Session, projection: Option<&Vec>, filters: &[Expr], limit: Option, diff --git a/src/table/src/table/numbers.rs b/src/table/src/table/numbers.rs index b3a1cc0ab0..82c19353a1 100644 --- a/src/table/src/table/numbers.rs +++ b/src/table/src/table/numbers.rs @@ -105,10 +105,16 @@ impl NumbersDataSource { impl DataSource for NumbersDataSource { fn get_stream(&self, request: ScanRequest) -> Result { + let projected_schema = match &request.projection { + Some(projection) => Arc::new(self.schema.try_project(projection).unwrap()), + None => self.schema.clone(), + }; Ok(Box::pin(NumbersStream { limit: request.limit.unwrap_or(100) as u32, schema: self.schema.clone(), already_run: false, + projection: request.projection, + projected_schema, })) } } @@ -118,11 +124,13 @@ struct NumbersStream { limit: u32, schema: SchemaRef, already_run: bool, + projection: Option>, + projected_schema: SchemaRef, } impl RecordBatchStream for NumbersStream { fn schema(&self) -> SchemaRef { - self.schema.clone() + self.projected_schema.clone() } fn output_ordering(&self) -> Option<&[OrderOption]> { @@ -143,14 +151,17 @@ impl Stream for NumbersStream { } self.already_run = true; let numbers: Vec = (0..self.limit).collect(); - let batch = DfRecordBatch::try_new( + let mut batch = DfRecordBatch::try_new( self.schema.arrow_schema().clone(), vec![Arc::new(UInt32Array::from(numbers))], ) .unwrap(); + if let Some(projection) = &self.projection { + batch = batch.project(projection).unwrap(); + } Poll::Ready(Some(RecordBatch::try_from_df_record_batch( - self.schema.clone(), + self.projected_schema.clone(), batch, ))) } diff --git a/src/table/src/table/scan.rs b/src/table/src/table/scan.rs index e4b47fa4fb..04424cf5a1 100644 --- a/src/table/src/table/scan.rs +++ b/src/table/src/table/scan.rs @@ -247,6 +247,10 @@ impl ExecutionPlan for RegionScanExec { }; Ok(statistics) } + + fn name(&self) -> &str { + "RegionScanExec" + } } impl DisplayAs for RegionScanExec { diff --git a/tests-integration/Cargo.toml b/tests-integration/Cargo.toml index d3434908b7..9f70eafc46 100644 --- a/tests-integration/Cargo.toml +++ b/tests-integration/Cargo.toml @@ -45,7 +45,8 @@ flow.workspace = true frontend = { workspace = true, features = ["testing"] } futures.workspace = true futures-util.workspace = true -loki-api = "0.1" +hyper-util = { workspace = true, features = ["tokio"] } +loki-api.workspace = true meta-client.workspace = true meta-srv = { workspace = true, features = ["mock"] } moka.workspace = true diff --git a/tests-integration/src/cluster.rs b/tests-integration/src/cluster.rs index 93664c2f19..d0051db7c5 100644 --- a/tests-integration/src/cluster.rs +++ b/tests-integration/src/cluster.rs @@ -49,6 +49,7 @@ use frontend::frontend::FrontendOptions; use frontend::heartbeat::HeartbeatTask; use frontend::instance::builder::FrontendBuilder; use frontend::instance::{FrontendInstance, Instance as FeInstance}; +use hyper_util::rt::TokioIo; use meta_client::client::MetaClientBuilder; use meta_srv::cluster::MetaPeerClientRef; use meta_srv::metasrv::{Metasrv, MetasrvOptions, SelectorRef}; @@ -486,7 +487,7 @@ async fn create_datanode_client(datanode: &Datanode) -> (String, Client) { async move { if let Some(client) = client { - Ok(client) + Ok(TokioIo::new(client)) } else { Err(std::io::Error::new( std::io::ErrorKind::Other, diff --git a/tests-integration/src/otlp.rs b/tests-integration/src/otlp.rs index b90b07b273..ed49522383 100644 --- a/tests-integration/src/otlp.rs +++ b/tests-integration/src/otlp.rs @@ -189,12 +189,14 @@ mod test { name: "my.test.metric".into(), description: "my ignored desc".into(), unit: "my ignored unit".into(), + metadata: vec![], data: Some(metric::Data::Gauge(gauge)), }, Metric { name: "my.test.histo".into(), description: "my ignored desc".into(), unit: "my ignored unit".into(), + metadata: vec![], data: Some(metric::Data::Histogram(histo)), }, ], diff --git a/tests-integration/src/tests/promql_test.rs b/tests-integration/src/tests/promql_test.rs index 3764e63ddf..1352531119 100644 --- a/tests-integration/src/tests/promql_test.rs +++ b/tests-integration/src/tests/promql_test.rs @@ -272,7 +272,7 @@ async fn aggregators_simple_sum(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+---------------------+--------------------------+\ - \n| group | ts | SUM(http_requests.value) |\ + \n| group | ts | sum(http_requests.value) |\ \n+------------+---------------------+--------------------------+\ \n| production | 1970-01-01T00:00:00 | 300.0 |\ \n| canary | 1970-01-01T00:00:00 | 700.0 |\ @@ -299,7 +299,7 @@ async fn aggregators_simple_avg(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+---------------------+--------------------------+\ - \n| group | ts | AVG(http_requests.value) |\ + \n| group | ts | avg(http_requests.value) |\ \n+------------+---------------------+--------------------------+\ \n| production | 1970-01-01T00:00:00 | 150.0 |\ \n| canary | 1970-01-01T00:00:00 | 350.0 |\ @@ -326,7 +326,7 @@ async fn aggregators_simple_count(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+---------------------+----------------------------+\ - \n| group | ts | COUNT(http_requests.value) |\ + \n| group | ts | count(http_requests.value) |\ \n+------------+---------------------+----------------------------+\ \n| canary | 1970-01-01T00:00:00 | 2 |\ \n| production | 1970-01-01T00:00:00 | 2 |\ @@ -353,7 +353,7 @@ async fn aggregators_simple_without(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+------------+---------------------+--------------------------+\ - \n| group | job | ts | SUM(http_requests.value) |\ + \n| group | job | ts | sum(http_requests.value) |\ \n+------------+------------+---------------------+--------------------------+\ \n| production | api-server | 1970-01-01T00:00:00 | 300.0 |\ \n| canary | api-server | 1970-01-01T00:00:00 | 700.0 |\ @@ -379,7 +379,7 @@ async fn aggregators_empty_by(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+---------------------+--------------------------+\ - \n| ts | SUM(http_requests.value) |\ + \n| ts | sum(http_requests.value) |\ \n+---------------------+--------------------------+\ \n| 1970-01-01T00:00:00 | 1000.0 |\ \n+---------------------+--------------------------+", @@ -404,7 +404,7 @@ async fn aggregators_no_by_without(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+---------------------+--------------------------+\ - \n| ts | SUM(http_requests.value) |\ + \n| ts | sum(http_requests.value) |\ \n+---------------------+--------------------------+\ \n| 1970-01-01T00:00:00 | 1000.0 |\ \n+---------------------+--------------------------+", @@ -430,7 +430,7 @@ async fn aggregators_empty_without(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+----------+------------+---------------------+--------------------------+\ - \n| group | instance | job | ts | SUM(http_requests.value) |\ + \n| group | instance | job | ts | sum(http_requests.value) |\ \n+------------+----------+------------+---------------------+--------------------------+\ \n| production | 0 | api-server | 1970-01-01T00:00:00 | 100.0 |\ \n| production | 1 | api-server | 1970-01-01T00:00:00 | 200.0 |\ @@ -457,7 +457,7 @@ async fn aggregators_complex_combined_aggrs(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------+\ - \n| job | ts | lhs.rhs.lhs.SUM(http_requests.value) + rhs.MIN(http_requests.value) + http_requests.MAX(http_requests.value) + rhs.AVG(http_requests.value) |\ + \n| job | ts | lhs.rhs.lhs.sum(http_requests.value) + rhs.min(http_requests.value) + http_requests.max(http_requests.value) + rhs.avg(http_requests.value) |\ \n+------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------+\ \n| api-server | 1970-01-01T00:00:00 | 1750.0 |\ \n| app-server | 1970-01-01T00:00:00 | 4550.0 |\ @@ -481,7 +481,7 @@ async fn two_aggregators_combined_aggrs(instance: Arc) { Duration::from_secs(60), Duration::from_secs(0), "+------------+---------------------+-------------------------------------------------------------+\ - \n| job | ts | lhs.SUM(http_requests.value) + rhs.MIN(http_requests.value) |\ + \n| job | ts | lhs.sum(http_requests.value) + rhs.min(http_requests.value) |\ \n+------------+---------------------+-------------------------------------------------------------+\ \n| api-server | 1970-01-01T00:00:00 | 1100.0 |\ \n| app-server | 1970-01-01T00:00:00 | 3100.0 |\ diff --git a/tests-integration/tests/http.rs b/tests-integration/tests/http.rs index 6378a659bf..8e1c4e116b 100644 --- a/tests-integration/tests/http.rs +++ b/tests-integration/tests/http.rs @@ -27,7 +27,6 @@ use loki_api::prost_types::Timestamp; use opentelemetry_proto::tonic::collector::logs::v1::ExportLogsServiceRequest; use opentelemetry_proto::tonic::collector::metrics::v1::ExportMetricsServiceRequest; use opentelemetry_proto::tonic::collector::trace::v1::ExportTraceServiceRequest; -use opentelemetry_proto::tonic::metrics::v1::ResourceMetrics; use prost::Message; use serde_json::{json, Value}; use servers::http::handler::HealthResponse; @@ -118,7 +117,7 @@ pub async fn test_http_auth(store_type: StorageType) { Some(user_provider), ) .await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // 1. no auth let res = client @@ -150,7 +149,7 @@ pub async fn test_http_auth(store_type: StorageType) { pub async fn test_sql_api(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "sql_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res = client.get("/v1/sql").send().await; assert_eq!(res.status(), StatusCode::BAD_REQUEST); @@ -413,7 +412,7 @@ pub async fn test_sql_api(store_type: StorageType) { pub async fn test_prometheus_promql_api(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "sql_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res = client .get("/v1/promql?query=abs(demo{host=\"Hangzhou\"})&start=0&end=100&step=5s") @@ -428,7 +427,7 @@ pub async fn test_prometheus_promql_api(store_type: StorageType) { pub async fn test_prom_http_api(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, mut guard) = setup_test_prom_app_with_frontend(store_type, "promql_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // format_query let res = client @@ -729,7 +728,7 @@ pub async fn test_prom_http_api(store_type: StorageType) { pub async fn test_metrics_api(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, mut guard) = setup_test_http_app(store_type, "metrics_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // Send a sql let res = client @@ -750,7 +749,7 @@ pub async fn test_metrics_api(store_type: StorageType) { pub async fn test_health_api(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, _guard) = setup_test_http_app_with_frontend(store_type, "health_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; async fn health_api(client: &TestClient, endpoint: &str) { // we can call health api with both `GET` and `POST` method. @@ -777,7 +776,7 @@ pub async fn test_health_api(store_type: StorageType) { pub async fn test_status_api(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, _guard) = setup_test_http_app_with_frontend(store_type, "status_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res_get = client.get("/status").send().await; assert_eq!(res_get.status(), StatusCode::OK); @@ -794,7 +793,7 @@ pub async fn test_status_api(store_type: StorageType) { pub async fn test_config_api(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, _guard) = setup_test_http_app_with_frontend(store_type, "config_api").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res_get = client.get("/config").send().await; assert_eq!(res_get.status(), StatusCode::OK); @@ -1022,7 +1021,7 @@ fn drop_lines_with_inconsistent_results(input: String) -> String { pub async fn test_dashboard_path(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, _guard) = setup_test_http_app_with_frontend(store_type, "dashboard_path").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res_get = client.get("/dashboard").send().await; assert_eq!(res_get.status(), StatusCode::PERMANENT_REDIRECT); @@ -1043,7 +1042,7 @@ pub async fn test_prometheus_remote_write(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, mut guard) = setup_test_prom_app_with_frontend(store_type, "prometheus_remote_write").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // write snappy encoded data let write_request = WriteRequest { @@ -1071,7 +1070,7 @@ pub async fn test_vm_proto_remote_write(store_type: StorageType) { setup_test_prom_app_with_frontend(store_type, "vm_proto_remote_write").await; // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; let res = client .post("/v1/prometheus/write?get_vm_proto_version=1") .send() @@ -1129,7 +1128,7 @@ pub async fn test_pipeline_api(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_pipeline_api").await; // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; let body = r#" processors: @@ -1260,7 +1259,7 @@ pub async fn test_identify_pipeline(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_pipeline_api").await; // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; let body = r#"{"__time__":1453809242,"__topic__":"","__source__":"10.170.***.***","ip":"10.200.**.***","time":"26/Jan/2016:19:54:02 +0800","url":"POST/PutData?Category=YunOsAccountOpLog&AccessKeyId=&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=HTTP/1.1","status":"200","user-agent":"aliyun-sdk-java"} {"__time__":1453809242,"__topic__":"","__source__":"10.170.***.***","ip":"10.200.**.***","time":"26/Jan/2016:19:54:02 +0800","url":"POST/PutData?Category=YunOsAccountOpLog&AccessKeyId=&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=HTTP/1.1","status":"200","user-agent":"aliyun-sdk-java","hasagei":"hasagei","dongdongdong":"guaguagua"}"#; let res = client @@ -1315,7 +1314,7 @@ pub async fn test_test_pipeline_api(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_pipeline_api").await; // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; let pipeline_content = r#" processors: @@ -1559,7 +1558,7 @@ pub async fn test_plain_text_ingestion(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_pipeline_api").await; // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; let body = r#" processors: @@ -1648,17 +1647,14 @@ pub async fn test_otlp_metrics(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_otlp_metrics").await; let content = r#" -{"resource":{"attributes":[],"droppedAttributesCount":0},"scopeMetrics":[{"scope":{"name":"","version":"","attributes":[],"droppedAttributesCount":0},"metrics":[{"name":"gen","description":"","unit":"","data":{"gauge":{"dataPoints":[{"attributes":[],"startTimeUnixNano":0,"timeUnixNano":1726053452870391000,"exemplars":[],"flags":0,"value":{"asInt":9471}}]}}}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.13.0"} +{"resourceMetrics":[{"resource":{"attributes":[],"droppedAttributesCount":0},"scopeMetrics":[{"scope":{"name":"","version":"","attributes":[],"droppedAttributesCount":0},"metrics":[{"name":"gen","description":"","unit":"","metadata":[],"gauge":{"dataPoints":[{"attributes":[],"startTimeUnixNano":"0","timeUnixNano":"1736489291872539000","exemplars":[],"flags":0,"asInt":0}]}}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.13.0"},{"resource":{"attributes":[],"droppedAttributesCount":0},"scopeMetrics":[{"scope":{"name":"","version":"","attributes":[],"droppedAttributesCount":0},"metrics":[{"name":"gen","description":"","unit":"","metadata":[],"gauge":{"dataPoints":[{"attributes":[],"startTimeUnixNano":"0","timeUnixNano":"1736489291919917000","exemplars":[],"flags":0,"asInt":1}]}}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.13.0"}]} "#; - let metrics: ResourceMetrics = serde_json::from_str(content).unwrap(); - let req = ExportMetricsServiceRequest { - resource_metrics: vec![metrics], - }; + let req: ExportMetricsServiceRequest = serde_json::from_str(content).unwrap(); let body = req.encode_to_vec(); // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; // write metrics data let res = send_req( @@ -1675,7 +1671,7 @@ pub async fn test_otlp_metrics(store_type: StorageType) { assert_eq!(StatusCode::OK, res.status()); // select metrics data - let expected = r#"[[1726053452870391000,9471.0]]"#; + let expected = "[[1736489291872539000,0.0],[1736489291919917000,1.0]]"; validate_data("otlp_metrics", &client, "select * from gen;", expected).await; // drop table @@ -1714,14 +1710,14 @@ pub async fn test_otlp_traces(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_otlp_traces").await; let content = r#" -{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"telemetrygen"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"telemetrygen","version":"","attributes":[],"droppedAttributesCount":0},"spans":[{"traceId":"b5e5fb572cf0a3335dd194a14145fef5","spanId":"74c82efa6f628e80","traceState":"","parentSpanId":"3364d2da58c9fd2b","flags":0,"name":"okey-dokey-0","kind":2,"startTimeUnixNano":1726631197820927000,"endTimeUnixNano":1726631197821050000,"attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-client"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}},{"traceId":"b5e5fb572cf0a3335dd194a14145fef5","spanId":"3364d2da58c9fd2b","traceState":"","parentSpanId":"","flags":0,"name":"lets-go","kind":3,"startTimeUnixNano":1726631197820927000,"endTimeUnixNano":1726631197821050000,"attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-server"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.4.0"}]} - "#; +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"telemetrygen"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"telemetrygen","version":"","attributes":[],"droppedAttributesCount":0},"spans":[{"traceId":"c05d7a4ec8e1f231f02ed6e8da8655b4","spanId":"9630f2916e2f7909","traceState":"","parentSpanId":"d24f921c75f68e23","flags":256,"name":"okey-dokey-0","kind":2,"startTimeUnixNano":"1736480942444376000","endTimeUnixNano":"1736480942444499000","attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-client"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}},{"traceId":"c05d7a4ec8e1f231f02ed6e8da8655b4","spanId":"d24f921c75f68e23","traceState":"","parentSpanId":"","flags":256,"name":"lets-go","kind":3,"startTimeUnixNano":"1736480942444376000","endTimeUnixNano":"1736480942444499000","attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-server"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}},{"traceId":"cc9e0991a2e63d274984bd44ee669203","spanId":"8f847259b0f6e1ab","traceState":"","parentSpanId":"eba7be77e3558179","flags":256,"name":"okey-dokey-0","kind":2,"startTimeUnixNano":"1736480942444589000","endTimeUnixNano":"1736480942444712000","attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-client"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}},{"traceId":"cc9e0991a2e63d274984bd44ee669203","spanId":"eba7be77e3558179","traceState":"","parentSpanId":"","flags":256,"name":"lets-go","kind":3,"startTimeUnixNano":"1736480942444589000","endTimeUnixNano":"1736480942444712000","attributes":[{"key":"net.peer.ip","value":{"stringValue":"1.2.3.4"}},{"key":"peer.service","value":{"stringValue":"telemetrygen-server"}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"links":[],"droppedLinksCount":0,"status":{"message":"","code":0}}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.4.0"}]} +"#; let req: ExportTraceServiceRequest = serde_json::from_str(content).unwrap(); let body = req.encode_to_vec(); // handshake - let client = TestClient::new(app); + let client = TestClient::new(app).await; // write traces data let res = send_req( @@ -1738,7 +1734,7 @@ pub async fn test_otlp_traces(store_type: StorageType) { assert_eq!(StatusCode::OK, res.status()); // select traces data - let expected = r#"[[1726631197820927000,1726631197821050000,123000,"telemetrygen","b5e5fb572cf0a3335dd194a14145fef5","3364d2da58c9fd2b","","SPAN_KIND_CLIENT","lets-go","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-server"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}],[1726631197820927000,1726631197821050000,123000,"telemetrygen","b5e5fb572cf0a3335dd194a14145fef5","74c82efa6f628e80","3364d2da58c9fd2b","SPAN_KIND_SERVER","okey-dokey-0","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-client"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}]]"#; + let expected = r#"[[1736480942444376000,1736480942444499000,123000,"telemetrygen","c05d7a4ec8e1f231f02ed6e8da8655b4","9630f2916e2f7909","d24f921c75f68e23","SPAN_KIND_SERVER","okey-dokey-0","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-client"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}],[1736480942444376000,1736480942444499000,123000,"telemetrygen","c05d7a4ec8e1f231f02ed6e8da8655b4","d24f921c75f68e23","","SPAN_KIND_CLIENT","lets-go","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-server"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}],[1736480942444589000,1736480942444712000,123000,"telemetrygen","cc9e0991a2e63d274984bd44ee669203","8f847259b0f6e1ab","eba7be77e3558179","SPAN_KIND_SERVER","okey-dokey-0","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-client"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}],[1736480942444589000,1736480942444712000,123000,"telemetrygen","cc9e0991a2e63d274984bd44ee669203","eba7be77e3558179","","SPAN_KIND_CLIENT","lets-go","STATUS_CODE_UNSET","","",{"net.peer.ip":"1.2.3.4","peer.service":"telemetrygen-server"},[],[],"telemetrygen","",{},{"service.name":"telemetrygen"}]]"#; validate_data( "otlp_traces", &client, @@ -1784,9 +1780,9 @@ pub async fn test_otlp_logs(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_otlp_logs").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let content = r#" -{"resourceLogs":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"schemaUrl":"https://opentelemetry.io/schemas/1.0.0/resourceLogs","scopeLogs":[{"scope":{},"schemaUrl":"https://opentelemetry.io/schemas/1.0.0/scopeLogs","logRecords":[{"flags":1,"timeUnixNano":1581452773000009875,"observedTimeUnixNano":1581452773000009875,"severityNumber":9,"severityText":"Info","body":{"value":{"stringValue":"This is a log message"}},"attributes":[{"key":"app","value":{"stringValue":"server"}},{"key":"instance_num","value":{"intValue":1}}],"droppedAttributesCount":1,"traceId":[48,56,48,52,48,50,48,49,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48,48],"spanId":[48,49,48,50,48,52,48,56,48,48,48,48,48,48,48,48]},{"flags":1,"timeUnixNano":1581452773000000789,"observedTimeUnixNano":1581452773000000789,"severityNumber":9,"severityText":"Info","body":{"value":{"stringValue":"something happened"}},"attributes":[{"key":"customer","value":{"stringValue":"acme"}},{"key":"env","value":{"stringValue":"dev"}}],"droppedAttributesCount":1,"traceId":[48],"spanId":[48]}]}]}]} +{"resourceLogs":[{"resource":{"attributes":[],"droppedAttributesCount":0},"scopeLogs":[{"scope":{"name":"","version":"","attributes":[],"droppedAttributesCount":0},"logRecords":[{"timeUnixNano":"1736413568497632000","observedTimeUnixNano":"0","severityNumber":9,"severityText":"Info","body":{"stringValue":"the message line one"},"attributes":[{"key":"app","value":{"stringValue":"server"}}],"droppedAttributesCount":0,"flags":0,"traceId":"f665100a612542b69cc362fe2ae9d3bf","spanId":"e58f01c4c69f4488"}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.4.0"},{"resource":{"attributes":[],"droppedAttributesCount":0},"scopeLogs":[{"scope":{"name":"","version":"","attributes":[],"droppedAttributesCount":0},"logRecords":[{"timeUnixNano":"1736413568538897000","observedTimeUnixNano":"0","severityNumber":9,"severityText":"Info","body":{"stringValue":"the message line two"},"attributes":[{"key":"app","value":{"stringValue":"server"}}],"droppedAttributesCount":0,"flags":0,"traceId":"f665100a612542b69cc362fe2ae9d3bf","spanId":"e58f01c4c69f4488"}],"schemaUrl":""}],"schemaUrl":"https://opentelemetry.io/schemas/1.4.0"}]} "#; let req: ExportLogsServiceRequest = serde_json::from_str(content).unwrap(); @@ -1812,7 +1808,7 @@ pub async fn test_otlp_logs(store_type: StorageType) { ) .await; assert_eq!(StatusCode::OK, res.status()); - let expected = r#"[[1581452773000000789,"30","30","Info",9,"something happened",{"customer":"acme","env":"dev"},1,"","",{},"https://opentelemetry.io/schemas/1.0.0/scopeLogs",{"resource-attr":"resource-attr-val-1"},"https://opentelemetry.io/schemas/1.0.0/resourceLogs"],[1581452773000009875,"3038303430323031303030303030303030303030303030303030303030303030","30313032303430383030303030303030","Info",9,"This is a log message",{"app":"server","instance_num":1},1,"","",{},"https://opentelemetry.io/schemas/1.0.0/scopeLogs",{"resource-attr":"resource-attr-val-1"},"https://opentelemetry.io/schemas/1.0.0/resourceLogs"]]"#; + let expected = "[[1736413568497632000,\"f665100a612542b69cc362fe2ae9d3bf\",\"e58f01c4c69f4488\",\"Info\",9,\"the message line one\",{\"app\":\"server\"},0,\"\",\"\",{},\"\",{},\"https://opentelemetry.io/schemas/1.4.0\"],[1736413568538897000,\"f665100a612542b69cc362fe2ae9d3bf\",\"e58f01c4c69f4488\",\"Info\",9,\"the message line two\",{\"app\":\"server\"},0,\"\",\"\",{},\"\",{},\"https://opentelemetry.io/schemas/1.4.0\"]]"; validate_data("otlp_logs", &client, "select * from logs1;", expected).await; } @@ -1841,7 +1837,7 @@ pub async fn test_otlp_logs(store_type: StorageType) { .await; assert_eq!(StatusCode::OK, res.status()); - let expected = r#"[[1581452773000000789,"30","30","Info",9,"something happened",{"customer":"acme","env":"dev"},1,"","",{},"https://opentelemetry.io/schemas/1.0.0/scopeLogs",{"resource-attr":"resource-attr-val-1"},"https://opentelemetry.io/schemas/1.0.0/resourceLogs",null,null,"resource-attr-val-1"],[1581452773000009875,"3038303430323031303030303030303030303030303030303030303030303030","30313032303430383030303030303030","Info",9,"This is a log message",{"app":"server","instance_num":1},1,"","",{},"https://opentelemetry.io/schemas/1.0.0/scopeLogs",{"resource-attr":"resource-attr-val-1"},"https://opentelemetry.io/schemas/1.0.0/resourceLogs","server",1,"resource-attr-val-1"]]"#; + let expected = "[[1736413568497632000,\"f665100a612542b69cc362fe2ae9d3bf\",\"e58f01c4c69f4488\",\"Info\",9,\"the message line one\",{\"app\":\"server\"},0,\"\",\"\",{},\"\",{},\"https://opentelemetry.io/schemas/1.4.0\",\"server\"],[1736413568538897000,\"f665100a612542b69cc362fe2ae9d3bf\",\"e58f01c4c69f4488\",\"Info\",9,\"the message line two\",{\"app\":\"server\"},0,\"\",\"\",{},\"\",{},\"https://opentelemetry.io/schemas/1.4.0\",\"server\"]]"; validate_data( "otlp_logs_with_selector", &client, @@ -1858,7 +1854,7 @@ pub async fn test_loki_pb_logs(store_type: StorageType) { common_telemetry::init_default_ut_logging(); let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_loki_pb_logs").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // init loki request let req: PushRequest = PushRequest { @@ -1936,7 +1932,7 @@ pub async fn test_loki_json_logs(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_loki_json_logs").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let body = r#" { @@ -2005,7 +2001,7 @@ pub async fn test_elasticsearch_logs(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_elasticsearch_logs").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; let body = r#" {"create":{"_index":"test","_id":"1"}} @@ -2046,7 +2042,7 @@ pub async fn test_elasticsearch_logs_with_index(store_type: StorageType) { let (app, mut guard) = setup_test_http_app_with_frontend(store_type, "test_elasticsearch_logs_with_index").await; - let client = TestClient::new(app); + let client = TestClient::new(app).await; // It will write to test_index1 and test_index2(specified in the path). let body = r#" @@ -2103,7 +2099,7 @@ async fn validate_data(test_name: &str, client: &TestClient, sql: &str, expected let v = get_rows_from_output(&resp); assert_eq!( - v, expected, + expected, v, "validate {test_name} fail, expected: {expected}, actual: {v}" ); } diff --git a/tests/cases/distributed/explain/analyze.result b/tests/cases/distributed/explain/analyze.result index 76aac4fa13..16b367e0c6 100644 --- a/tests/cases/distributed/explain/analyze.result +++ b/tests/cases/distributed/explain/analyze.result @@ -19,6 +19,14 @@ VALUES Affected Rows: 3 +SELECT count(*) FROM system_metrics; + ++----------+ +| count(*) | ++----------+ +| 3 | ++----------+ + -- SQLNESS REPLACE (metrics.*) REDACTED -- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED -- SQLNESS REPLACE (-+) - @@ -32,9 +40,10 @@ explain analyze SELECT count(*) FROM system_metrics; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_AggregateExec: mode=Final, gby=[], aggr=[COUNT(system_REDACTED +| 1_| 0_|_ProjectionExec: expr=[count(system_REDACTED +|_|_|_AggregateExec: mode=Final, gby=[], aggr=[count(system_REDACTED |_|_|_CoalescePartitionsExec REDACTED -|_|_|_AggregateExec: mode=Partial, gby=[], aggr=[COUNT(system_REDACTED +|_|_|_AggregateExec: mode=Partial, gby=[], aggr=[count(system_REDACTED |_|_|_RepartitionExec: partitioning=REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=1 (1 memtable ranges, 0 file 0 ranges) REDACTED |_|_|_| diff --git a/tests/cases/distributed/explain/analyze.sql b/tests/cases/distributed/explain/analyze.sql index c068e1dc6a..bdf51cbca4 100644 --- a/tests/cases/distributed/explain/analyze.sql +++ b/tests/cases/distributed/explain/analyze.sql @@ -15,6 +15,8 @@ VALUES ("host2", "idc_a", 80.0, 70.3, 90.0, 1667446797450), ("host1", "idc_b", 50.0, 66.7, 40.6, 1667446797450); +SELECT count(*) FROM system_metrics; + -- SQLNESS REPLACE (metrics.*) REDACTED -- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED -- SQLNESS REPLACE (-+) - diff --git a/tests/cases/distributed/explain/analyze_append_table_count.result b/tests/cases/distributed/explain/analyze_append_table_count.result index d08f7627fb..6dba2ad0cd 100644 --- a/tests/cases/distributed/explain/analyze_append_table_count.result +++ b/tests/cases/distributed/explain/analyze_append_table_count.result @@ -17,6 +17,19 @@ WITH( Affected Rows: 0 +INSERT INTO `test_table` (`bytes`, `http_version`, `ip`, `method`, `path`, `status`, `user`, `timestamp`) +VALUES (1024, 'HTTP/1.1', '192.168.1.1', 'GET', '/index.html', 200, 'user1', 1667446797450); + +Affected Rows: 1 + +SELECT count(*) FROM test_table; + ++----------+ +| count(*) | ++----------+ +| 1 | ++----------+ + -- SQLNESS REPLACE (metrics.*) REDACTED -- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED -- SQLNESS REPLACE (-+) - @@ -30,7 +43,7 @@ EXPLAIN ANALYZE SELECT count(*) FROM test_table; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_ProjectionExec: expr=[0 as COUNT(test_table.timestamp)] REDACTED +| 1_| 0_|_ProjectionExec: expr=[1 as count(*)] REDACTED |_|_|_common_recordbatch::adapter::MetricCollector REDACTED |_|_|_| |_|_| Total rows: 1_| diff --git a/tests/cases/distributed/explain/analyze_append_table_count.sql b/tests/cases/distributed/explain/analyze_append_table_count.sql index a47fc99275..fcb2e33c61 100644 --- a/tests/cases/distributed/explain/analyze_append_table_count.sql +++ b/tests/cases/distributed/explain/analyze_append_table_count.sql @@ -16,6 +16,11 @@ WITH( append_mode = 'true' ); +INSERT INTO `test_table` (`bytes`, `http_version`, `ip`, `method`, `path`, `status`, `user`, `timestamp`) +VALUES (1024, 'HTTP/1.1', '192.168.1.1', 'GET', '/index.html', 200, 'user1', 1667446797450); + +SELECT count(*) FROM test_table; + -- SQLNESS REPLACE (metrics.*) REDACTED -- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED -- SQLNESS REPLACE (-+) - diff --git a/tests/cases/distributed/explain/join_10_tables.result b/tests/cases/distributed/explain/join_10_tables.result index 5fa0194c14..6a38a88032 100644 --- a/tests/cases/distributed/explain/join_10_tables.result +++ b/tests/cases/distributed/explain/join_10_tables.result @@ -73,8 +73,7 @@ limit 1; +-+-+ | plan_type_| plan_| +-+-+ -| logical_plan_| Limit: skip=0, fetch=1_| -|_|_Sort: t_1.ts DESC NULLS FIRST, fetch=1_| +| logical_plan_| Sort: t_1.ts DESC NULLS FIRST, fetch=1_| |_|_Inner Join: t_9.ts = t_10.ts, t_9.vin = t_10.vin_| |_|_Inner Join: t_8.ts = t_9.ts, t_8.vin = t_9.vin_| |_|_Inner Join: t_7.ts = t_8.ts, t_7.vin = t_8.vin_| @@ -96,8 +95,7 @@ limit 1; |_|_MergeScan [is_placeholder=false]_| |_|_MergeScan [is_placeholder=false]_| |_|_MergeScan [is_placeholder=false]_| -| physical_plan | GlobalLimitExec: skip=0, fetch=1_| -|_|_SortPreservingMergeExec: [ts@0 DESC], fetch=1_| +| physical_plan | SortPreservingMergeExec: [ts@0 DESC], fetch=1_| |_|_SortExec: TopK(fetch=1), expr=[ts@0 DESC], preserve_partitioning=[true]_| |_|_CoalesceBatchesExec: target_batch_size=8192_| |_|_REDACTED diff --git a/tests/cases/distributed/explain/order_by.result b/tests/cases/distributed/explain/order_by.result index 64ddc07be9..6553ea1b68 100644 --- a/tests/cases/distributed/explain/order_by.result +++ b/tests/cases/distributed/explain/order_by.result @@ -61,8 +61,8 @@ EXPLAIN SELECT DISTINCT a, b FROM test ORDER BY a, b; | logical_plan_| Sort: test.a ASC NULLS LAST, test.b ASC NULLS LAST_| |_|_Aggregate: groupBy=[[test.a, test.b]], aggr=[[]]_| |_|_MergeScan [is_placeholder=false]_| -| physical_plan | SortPreservingMergeExec: [a@0 ASC NULLS LAST,b@1 ASC NULLS LAST]_| -|_|_SortExec: expr=[a@0 ASC NULLS LAST,b@1 ASC NULLS LAST], preserve_partitioning=[true] | +| physical_plan | SortPreservingMergeExec: [a@0 ASC NULLS LAST, b@1 ASC NULLS LAST]_| +|_|_SortExec: expr=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], preserve_partitioning=[true] | |_|_AggregateExec: mode=FinalPartitioned, gby=[a@0 as a, b@1 as b], aggr=[]_| |_|_CoalesceBatchesExec: target_batch_size=8192_| |_|_RepartitionExec: partitioning=REDACTED diff --git a/tests/cases/distributed/explain/subqueries.result b/tests/cases/distributed/explain/subqueries.result index 1a9ccd4e36..aff0ffc4b9 100644 --- a/tests/cases/distributed/explain/subqueries.result +++ b/tests/cases/distributed/explain/subqueries.result @@ -82,7 +82,7 @@ order by t.i desc; +-+-+ | logical_plan_| Sort: t.i DESC NULLS FIRST_| |_|_SubqueryAlias: t_| -|_|_CrossJoin:_| +|_|_Cross Join:_| |_|_Filter: integers.i IS NOT NULL_| |_|_Projection: integers.i_| |_|_MergeScan [is_placeholder=false]_| @@ -101,19 +101,26 @@ order by t.i desc; |_|_| +-+-+ +INSERT INTO other SELECT i, 2 FROM integers WHERE i=(SELECT MAX(i) FROM integers); + +Affected Rows: 0 + +-- Explain physical plan for DML is not supported because it looks up the table name in a way that is +-- different from normal queries. It also requires the table provider to implement the `insert_into()` method. EXPLAIN INSERT INTO other SELECT i, 2 FROM integers WHERE i=(SELECT MAX(i) FROM integers); -+--------------+-------------------------------------------------------------------+ -| plan_type | plan | -+--------------+-------------------------------------------------------------------+ -| logical_plan | Dml: op=[Insert Into] table=[other] | -| | Projection: integers.i AS i, TimestampMillisecond(2, None) AS j | -| | Inner Join: integers.i = __scalar_sq_1.MAX(integers.i) | -| | Projection: integers.i | -| | MergeScan [is_placeholder=false] | -| | SubqueryAlias: __scalar_sq_1 | -| | MergeScan [is_placeholder=false] | -+--------------+-------------------------------------------------------------------+ ++---------------------+-------------------------------------------------------------------+ +| plan_type | plan | ++---------------------+-------------------------------------------------------------------+ +| logical_plan | Dml: op=[Insert Into] table=[other] | +| | Projection: integers.i AS i, TimestampMillisecond(2, None) AS j | +| | Inner Join: integers.i = __scalar_sq_1.max(integers.i) | +| | Projection: integers.i | +| | MergeScan [is_placeholder=false] | +| | SubqueryAlias: __scalar_sq_1 | +| | MergeScan [is_placeholder=false] | +| physical_plan_error | Error during planning: failed to resolve catalog: datafusion | ++---------------------+-------------------------------------------------------------------+ drop table other; diff --git a/tests/cases/distributed/explain/subqueries.sql b/tests/cases/distributed/explain/subqueries.sql index deeb9cf283..0d1c6ff7af 100644 --- a/tests/cases/distributed/explain/subqueries.sql +++ b/tests/cases/distributed/explain/subqueries.sql @@ -28,6 +28,10 @@ from ( where t.i is not null order by t.i desc; +INSERT INTO other SELECT i, 2 FROM integers WHERE i=(SELECT MAX(i) FROM integers); + +-- Explain physical plan for DML is not supported because it looks up the table name in a way that is +-- different from normal queries. It also requires the table provider to implement the `insert_into()` method. EXPLAIN INSERT INTO other SELECT i, 2 FROM integers WHERE i=(SELECT MAX(i) FROM integers); drop table other; diff --git a/tests/cases/distributed/optimizer/filter_push_down.result b/tests/cases/distributed/optimizer/filter_push_down.result index 6adff4ba76..c9744b76be 100644 --- a/tests/cases/distributed/optimizer/filter_push_down.result +++ b/tests/cases/distributed/optimizer/filter_push_down.result @@ -228,7 +228,7 @@ SELECT i FROM (SELECT * FROM integers i1 UNION SELECT * FROM integers i2) a WHER SELECT * FROM (SELECT i1.i AS a, i2.i AS b, row_number() OVER (ORDER BY i1.i, i2.i) FROM integers i1, integers i2 WHERE i1.i IS NOT NULL AND i2.i IS NOT NULL) a1 WHERE a=b ORDER BY 1; +---+---+--------------------------------------------------------------------------------------------------------------------+ -| a | b | ROW_NUMBER() ORDER BY [i1.i ASC NULLS LAST, i2.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| a | b | row_number() ORDER BY [i1.i ASC NULLS LAST, i2.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +---+---+--------------------------------------------------------------------------------------------------------------------+ | 1 | 1 | 1 | | 2 | 2 | 5 | @@ -249,7 +249,7 @@ EXPLAIN SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2) a1 WHER | logical_plan_| Sort: a1.cond ASC NULLS LAST_| |_|_SubqueryAlias: a1_| |_|_Projection: Boolean(false) AS cond_| -|_|_CrossJoin:_| +|_|_Cross Join:_| |_|_SubqueryAlias: i1_| |_|_Projection:_| |_|_MergeScan [is_placeholder=false]_| @@ -267,9 +267,15 @@ EXPLAIN SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2) a1 WHER |_|_| +-+-+ +-- This should be a bug in DataFusion, it use UserDefinedLogicalNode::prevent_predicate_push_down_columns() +-- to prevent pushdown, but this filter is Literal(Boolean(false)). It doesn't reference any column. SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2 GROUP BY 1) a1 WHERE cond ORDER BY 1; -Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Attempted to create Filter predicate with expression `Boolean(false)` aliased as 'Int64(0) = Int64(1)'. Filter predicates should not be aliased. ++-------+ +| cond | ++-------+ +| false | ++-------+ DROP TABLE integers; diff --git a/tests/cases/distributed/optimizer/filter_push_down.sql b/tests/cases/distributed/optimizer/filter_push_down.sql index c34551f25e..a7f975385a 100644 --- a/tests/cases/distributed/optimizer/filter_push_down.sql +++ b/tests/cases/distributed/optimizer/filter_push_down.sql @@ -54,6 +54,8 @@ SELECT * FROM (SELECT i1.i AS a, i2.i AS b, row_number() OVER (ORDER BY i1.i, i2 -- SQLNESS REPLACE (peers.*) REDACTED EXPLAIN SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2) a1 WHERE cond ORDER BY 1; +-- This should be a bug in DataFusion, it use UserDefinedLogicalNode::prevent_predicate_push_down_columns() +-- to prevent pushdown, but this filter is Literal(Boolean(false)). It doesn't reference any column. SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2 GROUP BY 1) a1 WHERE cond ORDER BY 1; DROP TABLE integers; diff --git a/tests/cases/distributed/optimizer/order_by.result b/tests/cases/distributed/optimizer/order_by.result index 211dc40f6b..45ee66a6c5 100644 --- a/tests/cases/distributed/optimizer/order_by.result +++ b/tests/cases/distributed/optimizer/order_by.result @@ -36,26 +36,24 @@ explain select * from numbers order by number asc; -- SQLNESS REPLACE (peers.*) REDACTED explain select * from numbers order by number desc limit 10; -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| plan_type | plan | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| logical_plan | MergeScan [is_placeholder=false] | -| physical_plan | GlobalLimitExec: skip=0, fetch=10 | -| | SortExec: TopK(fetch=10), expr=[number@0 DESC], preserve_partitioning=[false] | -| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | -| | | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| plan_type | plan | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| logical_plan | MergeScan [is_placeholder=false] | +| physical_plan | SortExec: TopK(fetch=10), expr=[number@0 DESC], preserve_partitioning=[false] | +| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | +| | | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -- SQLNESS REPLACE (peers.*) REDACTED explain select * from numbers order by number asc limit 10; -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| plan_type | plan | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| logical_plan | MergeScan [is_placeholder=false] | -| physical_plan | GlobalLimitExec: skip=0, fetch=10 | -| | SortExec: TopK(fetch=10), expr=[number@0 ASC NULLS LAST], preserve_partitioning=[false] | -| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | -| | | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| plan_type | plan | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| logical_plan | MergeScan [is_placeholder=false] | +| physical_plan | SortExec: TopK(fetch=10), expr=[number@0 ASC NULLS LAST], preserve_partitioning=[false] | +| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | +| | | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/tests/cases/standalone/common/aggregate/count.result b/tests/cases/standalone/common/aggregate/count.result index f93189d985..0a8e96b357 100644 --- a/tests/cases/standalone/common/aggregate/count.result +++ b/tests/cases/standalone/common/aggregate/count.result @@ -9,7 +9,7 @@ Affected Rows: 2 select count(*) from "HelloWorld"; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 2 | +----------+ @@ -25,7 +25,7 @@ Affected Rows: 1 select count(*) from test; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 1 | +----------+ @@ -33,7 +33,7 @@ select count(*) from test; select count(*) from (select count(*) from test where a = 'a'); +----------+ -| COUNT(*) | +| count(*) | +----------+ | 1 | +----------+ @@ -41,7 +41,7 @@ select count(*) from (select count(*) from test where a = 'a'); select count(*) from (select * from test cross join "HelloWorld"); +----------+ -| COUNT(*) | +| count(*) | +----------+ | 2 | +----------+ @@ -56,15 +56,15 @@ Affected Rows: 0 -- Append table create table count_where_bug ( - tag String, + `tag` String, ts TimestampMillisecond time index, num Int64, - primary key (tag), + primary key (`tag`), ) engine=mito with('append_mode'='true'); Affected Rows: 0 -insert into count_where_bug (tag, ts, num) +insert into count_where_bug (`tag`, ts, num) values ('a', '2024-09-06T06:00:01Z', 1), ('a', '2024-09-06T06:00:02Z', 2), ('a', '2024-09-06T06:00:03Z', 3), @@ -73,10 +73,10 @@ values ('a', '2024-09-06T06:00:01Z', 1), Affected Rows: 5 -select count(1) from count_where_bug where tag = 'b'; +select count(1) from count_where_bug where `tag` = 'b'; +-----------------+ -| COUNT(Int64(1)) | +| count(Int64(1)) | +-----------------+ | 2 | +-----------------+ @@ -84,7 +84,7 @@ select count(1) from count_where_bug where tag = 'b'; select count(1) from count_where_bug where ts > '2024-09-06T06:00:04Z'; +-----------------+ -| COUNT(Int64(1)) | +| count(Int64(1)) | +-----------------+ | 1 | +-----------------+ @@ -92,7 +92,7 @@ select count(1) from count_where_bug where ts > '2024-09-06T06:00:04Z'; select count(1) from count_where_bug where num != 3; +-----------------+ -| COUNT(Int64(1)) | +| count(Int64(1)) | +-----------------+ | 4 | +-----------------+ diff --git a/tests/cases/standalone/common/aggregate/count.sql b/tests/cases/standalone/common/aggregate/count.sql index 22745b723c..dfc86f9273 100644 --- a/tests/cases/standalone/common/aggregate/count.sql +++ b/tests/cases/standalone/common/aggregate/count.sql @@ -21,20 +21,20 @@ drop table test; -- Append table create table count_where_bug ( - tag String, + `tag` String, ts TimestampMillisecond time index, num Int64, - primary key (tag), + primary key (`tag`), ) engine=mito with('append_mode'='true'); -insert into count_where_bug (tag, ts, num) +insert into count_where_bug (`tag`, ts, num) values ('a', '2024-09-06T06:00:01Z', 1), ('a', '2024-09-06T06:00:02Z', 2), ('a', '2024-09-06T06:00:03Z', 3), ('b', '2024-09-06T06:00:04Z', 4), ('b', '2024-09-06T06:00:05Z', 5); -select count(1) from count_where_bug where tag = 'b'; +select count(1) from count_where_bug where `tag` = 'b'; select count(1) from count_where_bug where ts > '2024-09-06T06:00:04Z'; diff --git a/tests/cases/standalone/common/aggregate/distinct.result b/tests/cases/standalone/common/aggregate/distinct.result index fea497c70a..cf0aaf1ad3 100644 --- a/tests/cases/standalone/common/aggregate/distinct.result +++ b/tests/cases/standalone/common/aggregate/distinct.result @@ -47,7 +47,7 @@ SELECT DISTINCT b FROM test ORDER BY b; SELECT DISTINCT a, SUM(B) FROM test GROUP BY a ORDER BY a; +----+-------------+ -| a | SUM(test.b) | +| a | sum(test.b) | +----+-------------+ | 11 | 65 | | 13 | 22 | @@ -56,7 +56,7 @@ SELECT DISTINCT a, SUM(B) FROM test GROUP BY a ORDER BY a; SELECT DISTINCT MAX(b) FROM test GROUP BY a; +-------------+ -| MAX(test.b) | +| max(test.b) | +-------------+ | 22 | +-------------+ diff --git a/tests/cases/standalone/common/aggregate/multi_regions.result b/tests/cases/standalone/common/aggregate/multi_regions.result index a7c6907eaf..ff72d4c2f9 100644 --- a/tests/cases/standalone/common/aggregate/multi_regions.result +++ b/tests/cases/standalone/common/aggregate/multi_regions.result @@ -27,21 +27,19 @@ select sum(val) from t group by host; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_ProjectionExec: expr=[SUM(t.val)@1 as SUM(t.val)] REDACTED -|_|_|_AggregateExec: mode=FinalPartitioned, gby=[host@0 as host], aggr=[SUM(t.val)] REDACTED +| 1_| 0_|_ProjectionExec: expr=[sum(t.val)@1 as sum(t.val)] REDACTED +|_|_|_AggregateExec: mode=FinalPartitioned, gby=[host@0 as host], aggr=[sum(t.val)] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_AggregateExec: mode=Partial, gby=[host@1 as host], aggr=[SUM(t.val)] REDACTED +|_|_|_AggregateExec: mode=Partial, gby=[host@1 as host], aggr=[sum(t.val)] REDACTED |_|_|_RepartitionExec: partitioning=REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=0 (0 memtable ranges, 0 file 0 ranges) REDACTED |_|_|_| -| 1_| 1_|_ProjectionExec: expr=[SUM(t.val)@1 as SUM(t.val)] REDACTED -|_|_|_AggregateExec: mode=FinalPartitioned, gby=[host@0 as host], aggr=[SUM(t.val)] REDACTED +| 1_| 1_|_ProjectionExec: expr=[sum(t.val)@1 as sum(t.val)] REDACTED +|_|_|_AggregateExec: mode=FinalPartitioned, gby=[host@0 as host], aggr=[sum(t.val)] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_AggregateExec: mode=Partial, gby=[host@1 as host], aggr=[SUM(t.val)] REDACTED +|_|_|_AggregateExec: mode=Partial, gby=[host@1 as host], aggr=[sum(t.val)] REDACTED |_|_|_RepartitionExec: partitioning=REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=0 (0 memtable ranges, 0 file 0 ranges) REDACTED |_|_|_| @@ -60,9 +58,9 @@ select sum(val) from t; +-+-+-+ | stage | node | plan_| +-+-+-+ -| 0_| 0_|_AggregateExec: mode=Final, gby=[], aggr=[SUM(t.val)] REDACTED +| 0_| 0_|_AggregateExec: mode=Final, gby=[], aggr=[sum(t.val)] REDACTED |_|_|_CoalescePartitionsExec REDACTED -|_|_|_AggregateExec: mode=Partial, gby=[], aggr=[SUM(t.val)] REDACTED +|_|_|_AggregateExec: mode=Partial, gby=[], aggr=[sum(t.val)] REDACTED |_|_|_ProjectionExec: expr=[val@1 as val] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| @@ -86,12 +84,11 @@ select sum(val) from t group by idc; +-+-+-+ | stage | node | plan_| +-+-+-+ -| 0_| 0_|_ProjectionExec: expr=[SUM(t.val)@1 as SUM(t.val)] REDACTED -|_|_|_AggregateExec: mode=FinalPartitioned, gby=[idc@0 as idc], aggr=[SUM(t.val)] REDACTED +| 0_| 0_|_ProjectionExec: expr=[sum(t.val)@1 as sum(t.val)] REDACTED +|_|_|_AggregateExec: mode=FinalPartitioned, gby=[idc@0 as idc], aggr=[sum(t.val)] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_AggregateExec: mode=Partial, gby=[idc@1 as idc], aggr=[SUM(t.val)] REDACTED +|_|_|_AggregateExec: mode=Partial, gby=[idc@1 as idc], aggr=[sum(t.val)] REDACTED |_|_|_ProjectionExec: expr=[val@1 as val, idc@3 as idc] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| diff --git a/tests/cases/standalone/common/aggregate/sum.result b/tests/cases/standalone/common/aggregate/sum.result index 5221e27761..1a82525520 100644 --- a/tests/cases/standalone/common/aggregate/sum.result +++ b/tests/cases/standalone/common/aggregate/sum.result @@ -1,7 +1,7 @@ SELECT SUM(number) FROM numbers; +---------------------+ -| SUM(numbers.number) | +| sum(numbers.number) | +---------------------+ | 4950 | +---------------------+ @@ -9,7 +9,7 @@ SELECT SUM(number) FROM numbers; SELECT SUM(1) FROM numbers; +---------------+ -| SUM(Int64(1)) | +| sum(Int64(1)) | +---------------+ | 100 | +---------------+ @@ -17,7 +17,7 @@ SELECT SUM(1) FROM numbers; SELECT SUM(-1) FROM numbers; +----------------+ -| SUM(Int64(-1)) | +| sum(Int64(-1)) | +----------------+ | -100 | +----------------+ @@ -25,7 +25,7 @@ SELECT SUM(-1) FROM numbers; SELECT SUM(-1) FROM numbers WHERE number=-1; +----------------+ -| SUM(Int64(-1)) | +| sum(Int64(-1)) | +----------------+ | | +----------------+ @@ -33,7 +33,7 @@ SELECT SUM(-1) FROM numbers WHERE number=-1; SELECT SUM(-1) FROM numbers WHERE number>10000 limit 1000; +----------------+ -| SUM(Int64(-1)) | +| sum(Int64(-1)) | +----------------+ | | +----------------+ @@ -49,7 +49,7 @@ Affected Rows: 3 SELECT SUM(b) FROM bigints; +----------------------+ -| SUM(bigints.b) | +| sum(bigints.b) | +----------------------+ | -9223372036854774807 | +----------------------+ @@ -65,7 +65,7 @@ Affected Rows: 4 SELECT sum(n) from doubles; +--------------------+ -| SUM(doubles.n) | +| sum(doubles.n) | +--------------------+ | 9007199254740992.0 | +--------------------+ diff --git a/tests/cases/standalone/common/alter/change_col_type.result b/tests/cases/standalone/common/alter/change_col_type.result index aa992a292c..95604c4164 100644 --- a/tests/cases/standalone/common/alter/change_col_type.result +++ b/tests/cases/standalone/common/alter/change_col_type.result @@ -1,4 +1,4 @@ -CREATE TABLE test(id INTEGER PRIMARY KEY, i INTEGER NULL, j TIMESTAMP TIME INDEX, k BOOLEAN); +CREATE TABLE test(`id` INTEGER PRIMARY KEY, i INTEGER NULL, j TIMESTAMP TIME INDEX, k BOOLEAN); Affected Rows: 0 diff --git a/tests/cases/standalone/common/alter/change_col_type.sql b/tests/cases/standalone/common/alter/change_col_type.sql index 3282385c87..ecaa1641b5 100644 --- a/tests/cases/standalone/common/alter/change_col_type.sql +++ b/tests/cases/standalone/common/alter/change_col_type.sql @@ -1,4 +1,4 @@ -CREATE TABLE test(id INTEGER PRIMARY KEY, i INTEGER NULL, j TIMESTAMP TIME INDEX, k BOOLEAN); +CREATE TABLE test(`id` INTEGER PRIMARY KEY, i INTEGER NULL, j TIMESTAMP TIME INDEX, k BOOLEAN); INSERT INTO test VALUES (1, 1, 1, false), (2, 2, 2, true); diff --git a/tests/cases/standalone/common/basic.result b/tests/cases/standalone/common/basic.result index a7a1dfb5c0..af6e24319b 100644 --- a/tests/cases/standalone/common/basic.result +++ b/tests/cases/standalone/common/basic.result @@ -32,7 +32,7 @@ SELECT * FROM system_metrics; SELECT count(*) FROM system_metrics; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 3 | +----------+ @@ -40,7 +40,7 @@ SELECT count(*) FROM system_metrics; SELECT avg(cpu_util) FROM system_metrics; +------------------------------+ -| AVG(system_metrics.cpu_util) | +| avg(system_metrics.cpu_util) | +------------------------------+ | 47.26666666666667 | +------------------------------+ @@ -48,7 +48,7 @@ SELECT avg(cpu_util) FROM system_metrics; SELECT idc, avg(memory_util) FROM system_metrics GROUP BY idc ORDER BY idc; +-------+---------------------------------+ -| idc | AVG(system_metrics.memory_util) | +| idc | avg(system_metrics.memory_util) | +-------+---------------------------------+ | idc_a | 40.3 | | idc_b | 66.7 | diff --git a/tests/cases/standalone/common/copy/copy_from_fs_parquet.result b/tests/cases/standalone/common/copy/copy_from_fs_parquet.result index 44e1422d5b..f31aebd7b1 100644 --- a/tests/cases/standalone/common/copy/copy_from_fs_parquet.result +++ b/tests/cases/standalone/common/copy/copy_from_fs_parquet.result @@ -98,7 +98,7 @@ Affected Rows: 6 select count(*) from without_limit_rows; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 6 | +----------+ @@ -114,14 +114,14 @@ Affected Rows: 3 select count(*) from with_limit_rows_segment; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 3 | +----------+ Copy with_limit_rows_segment FROM '${SQLNESS_HOME}/demo/export/parquet_files/' LIMIT hello; -Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'the number of maximum rows', found: ;: sql parser error: Expected literal int, found: hello at Line: 1, Column 86 +Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'the number of maximum rows', found: ;: sql parser error: Expected: literal int, found: hello at Line: 1, Column: 86 drop table demo; diff --git a/tests/cases/standalone/common/create/create_if_not_exists.result b/tests/cases/standalone/common/create/create_if_not_exists.result index 324196a683..a8288d0500 100644 --- a/tests/cases/standalone/common/create/create_if_not_exists.result +++ b/tests/cases/standalone/common/create/create_if_not_exists.result @@ -1,10 +1,10 @@ CREATE TABLE IF NOT EXIST t(); -Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'EXISTS', found: EXIST: sql parser error: Expected EXISTS, found: EXIST at Line: 1, Column 21 +Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'EXISTS', found: EXIST: sql parser error: Expected: EXISTS, found: EXIST at Line: 1, Column: 21 CREATE TABLE IF NOT t(); -Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'EXISTS', found: t: sql parser error: Expected EXISTS, found: t at Line: 1, Column 21 +Error: 2000(InvalidSyntax), Unexpected token while parsing SQL statement, expected: 'EXISTS', found: t: sql parser error: Expected: EXISTS, found: t at Line: 1, Column: 21 CREATE TABLE IF EXISTS t(); diff --git a/tests/cases/standalone/common/create/create_with_options.result b/tests/cases/standalone/common/create/create_with_options.result index 67015eeab9..8379451e11 100644 --- a/tests/cases/standalone/common/create/create_with_options.result +++ b/tests/cases/standalone/common/create/create_with_options.result @@ -1,5 +1,5 @@ CREATE TABLE not_supported_table_options_keys ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT, @@ -7,10 +7,10 @@ CREATE TABLE not_supported_table_options_keys ( TIME INDEX (ts), PRIMARY KEY (id, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( diff --git a/tests/cases/standalone/common/create/create_with_options.sql b/tests/cases/standalone/common/create/create_with_options.sql index f9a6d47905..3c848cd27e 100644 --- a/tests/cases/standalone/common/create/create_with_options.sql +++ b/tests/cases/standalone/common/create/create_with_options.sql @@ -1,5 +1,5 @@ CREATE TABLE not_supported_table_options_keys ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT, @@ -7,10 +7,10 @@ CREATE TABLE not_supported_table_options_keys ( TIME INDEX (ts), PRIMARY KEY (id, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( diff --git a/tests/cases/standalone/common/create/create_with_skipping_index.result b/tests/cases/standalone/common/create/create_with_skipping_index.result index 00dd24dc6c..8cbe4e53f0 100644 --- a/tests/cases/standalone/common/create/create_with_skipping_index.result +++ b/tests/cases/standalone/common/create/create_with_skipping_index.result @@ -1,7 +1,7 @@ create table skipping_table ( ts timestamp time index, - id string skipping index, + `id` string skipping index, `name` string skipping index with (granularity = 8192), diff --git a/tests/cases/standalone/common/create/create_with_skipping_index.sql b/tests/cases/standalone/common/create/create_with_skipping_index.sql index 0558936699..00330519de 100644 --- a/tests/cases/standalone/common/create/create_with_skipping_index.sql +++ b/tests/cases/standalone/common/create/create_with_skipping_index.sql @@ -1,7 +1,7 @@ create table skipping_table ( ts timestamp time index, - id string skipping index, + `id` string skipping index, `name` string skipping index with (granularity = 8192), diff --git a/tests/cases/standalone/common/cte/cte.result b/tests/cases/standalone/common/cte/cte.result index cf237fcb5a..c63ae6735f 100644 --- a/tests/cases/standalone/common/cte/cte.result +++ b/tests/cases/standalone/common/cte/cte.result @@ -97,7 +97,6 @@ SELECT 1 UNION ALL (WITH cte AS (SELECT 42) SELECT * FROM cte) order by 1; | 42 | +----------+ --- Recursive CTEs are not supported in datafusion WITH RECURSIVE cte(d) AS ( SELECT 1 UNION ALL @@ -109,7 +108,11 @@ WITH RECURSIVE cte(d) AS ( ) SELECT max(d) FROM cte; -Error: 1001(Unsupported), Failed to plan SQL: This feature is not implemented: Recursive CTE is not implemented ++------------+ +| max(cte.d) | ++------------+ +| 1 | ++------------+ -- Nested aliases is not supported in datafusion with cte (a) as ( diff --git a/tests/cases/standalone/common/cte/cte.sql b/tests/cases/standalone/common/cte/cte.sql index 8e1022a29b..0b190df11f 100644 --- a/tests/cases/standalone/common/cte/cte.sql +++ b/tests/cases/standalone/common/cte/cte.sql @@ -27,7 +27,6 @@ with cte1(x, y) as (select 42 a, 84 b) select zzz, y from cte1 t1(zzz, y); SELECT 1 UNION ALL (WITH cte AS (SELECT 42) SELECT * FROM cte) order by 1; --- Recursive CTEs are not supported in datafusion WITH RECURSIVE cte(d) AS ( SELECT 1 UNION ALL diff --git a/tests/cases/standalone/common/flow/flow_auto_sink_table.result b/tests/cases/standalone/common/flow/flow_auto_sink_table.result index e8bd5cf739..ebd19a828d 100644 --- a/tests/cases/standalone/common/flow/flow_auto_sink_table.result +++ b/tests/cases/standalone/common/flow/flow_auto_sink_table.result @@ -23,7 +23,7 @@ SHOW CREATE TABLE out_num_cnt_basic; | Table | Create Table | +-------------------+--------------------------------------------------+ | out_num_cnt_basic | CREATE TABLE IF NOT EXISTS "out_num_cnt_basic" ( | -| | "SUM(numbers_input_basic.number)" BIGINT NULL, | +| | "sum(numbers_input_basic.number)" BIGINT NULL, | | | "window_start" TIMESTAMP(3) NOT NULL, | | | "window_end" TIMESTAMP(3) NULL, | | | "update_at" TIMESTAMP(3) NULL, | @@ -51,7 +51,7 @@ SHOW CREATE TABLE out_num_cnt_basic; | Table | Create Table | +-------------------+--------------------------------------------------+ | out_num_cnt_basic | CREATE TABLE IF NOT EXISTS "out_num_cnt_basic" ( | -| | "SUM(numbers_input_basic.number)" BIGINT NULL, | +| | "sum(numbers_input_basic.number)" BIGINT NULL, | | | "window_start" TIMESTAMP(3) NOT NULL, | | | "window_end" TIMESTAMP(3) NULL, | | | "update_at" TIMESTAMP(3) NULL, | diff --git a/tests/cases/standalone/common/flow/flow_basic.result b/tests/cases/standalone/common/flow/flow_basic.result index e17efd74be..049ef70f30 100644 --- a/tests/cases/standalone/common/flow/flow_basic.result +++ b/tests/cases/standalone/common/flow/flow_basic.result @@ -23,7 +23,7 @@ SHOW CREATE TABLE out_num_cnt_basic; | Table | Create Table | +-------------------+--------------------------------------------------+ | out_num_cnt_basic | CREATE TABLE IF NOT EXISTS "out_num_cnt_basic" ( | -| | "SUM(numbers_input_basic.number)" BIGINT NULL, | +| | "sum(numbers_input_basic.number)" BIGINT NULL, | | | "window_start" TIMESTAMP(3) NOT NULL, | | | "window_end" TIMESTAMP(3) NULL, | | | "update_at" TIMESTAMP(3) NULL, | @@ -52,7 +52,7 @@ SHOW CREATE TABLE out_num_cnt_basic; | Table | Create Table | +-------------------+--------------------------------------------------+ | out_num_cnt_basic | CREATE TABLE IF NOT EXISTS "out_num_cnt_basic" ( | -| | "SUM(numbers_input_basic.number)" BIGINT NULL, | +| | "sum(numbers_input_basic.number)" BIGINT NULL, | | | "window_start" TIMESTAMP(3) NOT NULL, | | | "window_end" TIMESTAMP(3) NULL, | | | "update_at" TIMESTAMP(3) NULL, | @@ -83,14 +83,14 @@ ADMIN FLUSH_FLOW('test_numbers_basic'); +----------------------------------------+ SELECT - "SUM(numbers_input_basic.number)", + "sum(numbers_input_basic.number)", window_start, window_end FROM out_num_cnt_basic; +---------------------------------+---------------------+---------------------+ -| SUM(numbers_input_basic.number) | window_start | window_end | +| sum(numbers_input_basic.number) | window_start | window_end | +---------------------------------+---------------------+---------------------+ | 42 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | +---------------------------------+---------------------+---------------------+ @@ -123,14 +123,14 @@ ADMIN FLUSH_FLOW('test_numbers_basic'); -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion SELECT - "SUM(numbers_input_basic.number)", + "sum(numbers_input_basic.number)", window_start, window_end FROM out_num_cnt_basic; +---------------------------------+---------------------+---------------------+ -| SUM(numbers_input_basic.number) | window_start | window_end | +| sum(numbers_input_basic.number) | window_start | window_end | +---------------------------------+---------------------+---------------------+ | 42 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | | 47 | 2021-07-01T00:00:01 | 2021-07-01T00:00:02 | @@ -685,11 +685,11 @@ SHOW CREATE TABLE ngx_country; | Table | Create Table | +-------------+---------------------------------------------+ | ngx_country | CREATE TABLE IF NOT EXISTS "ngx_country" ( | -| | "ngx_access_log.country" STRING NULL, | +| | "country" STRING NULL, | | | "update_at" TIMESTAMP(3) NULL, | | | "__ts_placeholder" TIMESTAMP(3) NOT NULL, | | | TIME INDEX ("__ts_placeholder"), | -| | PRIMARY KEY ("ngx_access_log.country") | +| | PRIMARY KEY ("country") | | | ) | | | | | | ENGINE=mito | @@ -718,11 +718,11 @@ SHOW CREATE TABLE ngx_country; | Table | Create Table | +-------------+---------------------------------------------+ | ngx_country | CREATE TABLE IF NOT EXISTS "ngx_country" ( | -| | "ngx_access_log.country" STRING NULL, | +| | "country" STRING NULL, | | | "update_at" TIMESTAMP(3) NULL, | | | "__ts_placeholder" TIMESTAMP(3) NOT NULL, | | | TIME INDEX ("__ts_placeholder"), | -| | PRIMARY KEY ("ngx_access_log.country") | +| | PRIMARY KEY ("country") | | | ) | | | | | | ENGINE=mito | @@ -730,15 +730,15 @@ SHOW CREATE TABLE ngx_country; +-------------+---------------------------------------------+ SELECT - "ngx_access_log.country" + country FROM ngx_country; -+------------------------+ -| ngx_access_log.country | -+------------------------+ -| b | -+------------------------+ ++---------+ +| country | ++---------+ +| b | ++---------+ -- making sure distinct is working INSERT INTO @@ -758,15 +758,15 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.country" + country FROM ngx_country; -+------------------------+ -| ngx_access_log.country | -+------------------------+ -| b | -+------------------------+ ++---------+ +| country | ++---------+ +| b | ++---------+ INSERT INTO ngx_access_log @@ -785,16 +785,16 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.country" + country FROM ngx_country; -+------------------------+ -| ngx_access_log.country | -+------------------------+ -| b | -| c | -+------------------------+ ++---------+ +| country | ++---------+ +| b | +| c | ++---------+ DROP FLOW calc_ngx_country; @@ -835,11 +835,11 @@ SHOW CREATE TABLE ngx_country; | Table | Create Table | +-------------+--------------------------------------------+ | ngx_country | CREATE TABLE IF NOT EXISTS "ngx_country" ( | -| | "ngx_access_log.country" STRING NULL, | +| | "country" STRING NULL, | | | "time_window" TIMESTAMP(3) NOT NULL, | | | "update_at" TIMESTAMP(3) NULL, | | | TIME INDEX ("time_window"), | -| | PRIMARY KEY ("ngx_access_log.country") | +| | PRIMARY KEY ("country") | | | ) | | | | | | ENGINE=mito | @@ -868,11 +868,11 @@ SHOW CREATE TABLE ngx_country; | Table | Create Table | +-------------+--------------------------------------------+ | ngx_country | CREATE TABLE IF NOT EXISTS "ngx_country" ( | -| | "ngx_access_log.country" STRING NULL, | +| | "country" STRING NULL, | | | "time_window" TIMESTAMP(3) NOT NULL, | | | "update_at" TIMESTAMP(3) NULL, | | | TIME INDEX ("time_window"), | -| | PRIMARY KEY ("ngx_access_log.country") | +| | PRIMARY KEY ("country") | | | ) | | | | | | ENGINE=mito | @@ -880,16 +880,16 @@ SHOW CREATE TABLE ngx_country; +-------------+--------------------------------------------+ SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; -+------------------------+---------------------+ -| ngx_access_log.country | time_window | -+------------------------+---------------------+ -| b | 1970-01-01T00:00:00 | -+------------------------+---------------------+ ++---------+---------------------+ +| country | time_window | ++---------+---------------------+ +| b | 1970-01-01T00:00:00 | ++---------+---------------------+ -- making sure distinct is working INSERT INTO @@ -909,16 +909,16 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; -+------------------------+---------------------+ -| ngx_access_log.country | time_window | -+------------------------+---------------------+ -| b | 1970-01-01T00:00:00 | -+------------------------+---------------------+ ++---------+---------------------+ +| country | time_window | ++---------+---------------------+ +| b | 1970-01-01T00:00:00 | ++---------+---------------------+ INSERT INTO ngx_access_log @@ -937,17 +937,17 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; -+------------------------+---------------------+ -| ngx_access_log.country | time_window | -+------------------------+---------------------+ -| b | 1970-01-01T00:00:00 | -| c | 1970-01-01T00:00:00 | -+------------------------+---------------------+ ++---------+---------------------+ +| country | time_window | ++---------+---------------------+ +| b | 1970-01-01T00:00:00 | +| c | 1970-01-01T00:00:00 | ++---------+---------------------+ DROP FLOW calc_ngx_country; diff --git a/tests/cases/standalone/common/flow/flow_basic.sql b/tests/cases/standalone/common/flow/flow_basic.sql index 516afa4074..c5cf6d36a6 100644 --- a/tests/cases/standalone/common/flow/flow_basic.sql +++ b/tests/cases/standalone/common/flow/flow_basic.sql @@ -33,7 +33,7 @@ VALUES ADMIN FLUSH_FLOW('test_numbers_basic'); SELECT - "SUM(numbers_input_basic.number)", + "sum(numbers_input_basic.number)", window_start, window_end FROM @@ -53,7 +53,7 @@ ADMIN FLUSH_FLOW('test_numbers_basic'); -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion SELECT - "SUM(numbers_input_basic.number)", + "sum(numbers_input_basic.number)", window_start, window_end FROM @@ -326,7 +326,7 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); SHOW CREATE TABLE ngx_country; SELECT - "ngx_access_log.country" + country FROM ngx_country; @@ -340,7 +340,7 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.country" + country FROM ngx_country; @@ -353,7 +353,7 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.country" + country FROM ngx_country; @@ -393,7 +393,7 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); SHOW CREATE TABLE ngx_country; SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; @@ -408,7 +408,7 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; @@ -422,7 +422,7 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.country", + country, time_window FROM ngx_country; diff --git a/tests/cases/standalone/common/flow/flow_call_df_func.result b/tests/cases/standalone/common/flow/flow_call_df_func.result index a2b300796b..d6423c7c7f 100644 --- a/tests/cases/standalone/common/flow/flow_call_df_func.result +++ b/tests/cases/standalone/common/flow/flow_call_df_func.result @@ -42,10 +42,10 @@ ADMIN FLUSH_FLOW('test_numbers_df_func'); +------------------------------------------+ -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion -SELECT "SUM(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "sum(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +----------------------------------------+---------------------+---------------------+ -| SUM(abs(numbers_input_df_func.number)) | window_start | window_end | +| sum(abs(numbers_input_df_func.number)) | window_start | window_end | +----------------------------------------+---------------------+---------------------+ | 42 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | +----------------------------------------+---------------------+---------------------+ @@ -76,10 +76,10 @@ ADMIN FLUSH_FLOW('test_numbers_df_func'); +------------------------------------------+ -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion -SELECT "SUM(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "sum(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +----------------------------------------+---------------------+---------------------+ -| SUM(abs(numbers_input_df_func.number)) | window_start | window_end | +| sum(abs(numbers_input_df_func.number)) | window_start | window_end | +----------------------------------------+---------------------+---------------------+ | 42 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | | 47 | 2021-07-01T00:00:01 | 2021-07-01T00:00:02 | @@ -140,10 +140,10 @@ ADMIN FLUSH_FLOW('test_numbers_df_func'); | FLOW_FLUSHED | +------------------------------------------+ -SELECT "abs(SUM(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "abs(sum(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +----------------------------------------+---------------------+---------------------+ -| abs(SUM(numbers_input_df_func.number)) | window_start | window_end | +| abs(sum(numbers_input_df_func.number)) | window_start | window_end | +----------------------------------------+---------------------+---------------------+ | 2 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | +----------------------------------------+---------------------+---------------------+ @@ -173,10 +173,10 @@ ADMIN FLUSH_FLOW('test_numbers_df_func'); | FLOW_FLUSHED | +------------------------------------------+ -SELECT "abs(SUM(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "abs(sum(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +----------------------------------------+---------------------+---------------------+ -| abs(SUM(numbers_input_df_func.number)) | window_start | window_end | +| abs(sum(numbers_input_df_func.number)) | window_start | window_end | +----------------------------------------+---------------------+---------------------+ | 2 | 2021-07-01T00:00:00 | 2021-07-01T00:00:01 | | 1 | 2021-07-01T00:00:01 | 2021-07-01T00:00:02 | diff --git a/tests/cases/standalone/common/flow/flow_call_df_func.sql b/tests/cases/standalone/common/flow/flow_call_df_func.sql index a6e0030d05..6143f493f4 100644 --- a/tests/cases/standalone/common/flow/flow_call_df_func.sql +++ b/tests/cases/standalone/common/flow/flow_call_df_func.sql @@ -24,7 +24,7 @@ VALUES ADMIN FLUSH_FLOW('test_numbers_df_func'); -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion -SELECT "SUM(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "sum(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; -- SQLNESS REPLACE (ADMIN\sFLUSH_FLOW\('\w+'\)\s+\|\n\+-+\+\n\|\s+)[0-9]+\s+\| $1 FLOW_FLUSHED | ADMIN FLUSH_FLOW('test_numbers_df_func'); @@ -38,7 +38,7 @@ VALUES ADMIN FLUSH_FLOW('test_numbers_df_func'); -- note that this quote-unquote column is a column-name, **not** a aggregation expr, generated by datafusion -SELECT "SUM(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "sum(abs(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; DROP FLOW test_numbers_df_func; DROP TABLE numbers_input_df_func; @@ -69,7 +69,7 @@ VALUES -- SQLNESS REPLACE (ADMIN\sFLUSH_FLOW\('\w+'\)\s+\|\n\+-+\+\n\|\s+)[0-9]+\s+\| $1 FLOW_FLUSHED | ADMIN FLUSH_FLOW('test_numbers_df_func'); -SELECT "abs(SUM(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "abs(sum(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; -- SQLNESS REPLACE (ADMIN\sFLUSH_FLOW\('\w+'\)\s+\|\n\+-+\+\n\|\s+)[0-9]+\s+\| $1 FLOW_FLUSHED | ADMIN FLUSH_FLOW('test_numbers_df_func'); @@ -82,7 +82,7 @@ VALUES -- SQLNESS REPLACE (ADMIN\sFLUSH_FLOW\('\w+'\)\s+\|\n\+-+\+\n\|\s+)[0-9]+\s+\| $1 FLOW_FLUSHED | ADMIN FLUSH_FLOW('test_numbers_df_func'); -SELECT "abs(SUM(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; +SELECT "abs(sum(numbers_input_df_func.number))", window_start, window_end FROM out_num_cnt_df_func; DROP FLOW test_numbers_df_func; DROP TABLE numbers_input_df_func; diff --git a/tests/cases/standalone/common/flow/flow_null.result b/tests/cases/standalone/common/flow/flow_null.result index 63afaf33e2..9ec66eb619 100644 --- a/tests/cases/standalone/common/flow/flow_null.result +++ b/tests/cases/standalone/common/flow/flow_null.result @@ -134,21 +134,21 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM ngx_country; -+-----------------------+------------------------+---------------+---------------------+ -| ngx_access_log.client | ngx_access_log.country | country_count | time_window | -+-----------------------+------------------------+---------------+---------------------+ -| cli1 | | 2 | 1970-01-01T00:00:00 | -| cli1 | b | 1 | 1970-01-01T00:00:00 | -| cli1 | c | 1 | 1970-01-01T00:00:00 | -| cli2 | | 2 | 1970-01-01T00:00:00 | -+-----------------------+------------------------+---------------+---------------------+ ++--------+---------+---------------+---------------------+ +| client | country | country_count | time_window | ++--------+---------+---------------+---------------------+ +| cli1 | | 2 | 1970-01-01T00:00:00 | +| cli1 | b | 1 | 1970-01-01T00:00:00 | +| cli1 | c | 1 | 1970-01-01T00:00:00 | +| cli2 | | 2 | 1970-01-01T00:00:00 | ++--------+---------+---------------+---------------------+ -- making sure distinct is working INSERT INTO @@ -168,21 +168,21 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM ngx_country; -+-----------------------+------------------------+---------------+---------------------+ -| ngx_access_log.client | ngx_access_log.country | country_count | time_window | -+-----------------------+------------------------+---------------+---------------------+ -| cli1 | | 2 | 1970-01-01T00:00:00 | -| cli1 | b | 2 | 1970-01-01T00:00:00 | -| cli1 | c | 1 | 1970-01-01T00:00:00 | -| cli2 | | 2 | 1970-01-01T00:00:00 | -+-----------------------+------------------------+---------------+---------------------+ ++--------+---------+---------------+---------------------+ +| client | country | country_count | time_window | ++--------+---------+---------------+---------------------+ +| cli1 | | 2 | 1970-01-01T00:00:00 | +| cli1 | b | 2 | 1970-01-01T00:00:00 | +| cli1 | c | 1 | 1970-01-01T00:00:00 | +| cli2 | | 2 | 1970-01-01T00:00:00 | ++--------+---------+---------------+---------------------+ INSERT INTO ngx_access_log @@ -201,21 +201,21 @@ ADMIN FLUSH_FLOW('calc_ngx_country'); +--------------------------------------+ SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM ngx_country; -+-----------------------+------------------------+---------------+---------------------+ -| ngx_access_log.client | ngx_access_log.country | country_count | time_window | -+-----------------------+------------------------+---------------+---------------------+ -| cli1 | | 2 | 1970-01-01T00:00:00 | -| cli1 | b | 2 | 1970-01-01T00:00:00 | -| cli1 | c | 2 | 1970-01-01T00:00:00 | -| cli2 | | 2 | 1970-01-01T00:00:00 | -+-----------------------+------------------------+---------------+---------------------+ ++--------+---------+---------------+---------------------+ +| client | country | country_count | time_window | ++--------+---------+---------------+---------------------+ +| cli1 | | 2 | 1970-01-01T00:00:00 | +| cli1 | b | 2 | 1970-01-01T00:00:00 | +| cli1 | c | 2 | 1970-01-01T00:00:00 | +| cli2 | | 2 | 1970-01-01T00:00:00 | ++--------+---------+---------------+---------------------+ DROP FLOW calc_ngx_country; diff --git a/tests/cases/standalone/common/flow/flow_null.sql b/tests/cases/standalone/common/flow/flow_null.sql index 5de72d39bc..89ccd366cb 100644 --- a/tests/cases/standalone/common/flow/flow_null.sql +++ b/tests/cases/standalone/common/flow/flow_null.sql @@ -88,8 +88,8 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM @@ -105,8 +105,8 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM @@ -121,8 +121,8 @@ VALUES ADMIN FLUSH_FLOW('calc_ngx_country'); SELECT - "ngx_access_log.client", - "ngx_access_log.country", + client, + country, country_count, time_window FROM diff --git a/tests/cases/standalone/common/function/date.result b/tests/cases/standalone/common/function/date.result index 3fce12117f..e9c1ee570d 100644 --- a/tests/cases/standalone/common/function/date.result +++ b/tests/cases/standalone/common/function/date.result @@ -1,11 +1,11 @@ --- date_add --- SELECT date_add('2023-12-06 07:39:46.222'::TIMESTAMP_MS, INTERVAL '5 day'); -+-----------------------------------------------------------------------------------------------------------------------------------------+ -| date_add(arrow_cast(Utf8("2023-12-06 07:39:46.222"),Utf8("Timestamp(Millisecond, None)")),IntervalMonthDayNano("92233720368547758080")) | -+-----------------------------------------------------------------------------------------------------------------------------------------+ -| 2023-12-11T07:39:46.222 | -+-----------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| date_add(arrow_cast(Utf8("2023-12-06 07:39:46.222"),Utf8("Timestamp(Millisecond, None)")),IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 5, nanoseconds: 0 }")) | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-12-11T07:39:46.222 | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT date_add('2023-12-06 07:39:46.222'::TIMESTAMP_MS, '5 day'); @@ -17,11 +17,11 @@ SELECT date_add('2023-12-06 07:39:46.222'::TIMESTAMP_MS, '5 day'); SELECT date_add('2023-12-06'::DATE, INTERVAL '3 month 5 day'); -+-------------------------------------------------------------------------------------+ -| date_add(Utf8("2023-12-06"),IntervalMonthDayNano("237684487635026733149179609088")) | -+-------------------------------------------------------------------------------------+ -| 2024-03-11 | -+-------------------------------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------------------+ +| date_add(Utf8("2023-12-06"),IntervalMonthDayNano("IntervalMonthDayNano { months: 3, days: 5, nanoseconds: 0 }")) | ++------------------------------------------------------------------------------------------------------------------+ +| 2024-03-11 | ++------------------------------------------------------------------------------------------------------------------+ SELECT date_add('2023-12-06'::DATE, '3 month 5 day'); @@ -34,11 +34,11 @@ SELECT date_add('2023-12-06'::DATE, '3 month 5 day'); --- date_sub --- SELECT date_sub('2023-12-06 07:39:46.222'::TIMESTAMP_MS, INTERVAL '5 day'); -+-----------------------------------------------------------------------------------------------------------------------------------------+ -| date_sub(arrow_cast(Utf8("2023-12-06 07:39:46.222"),Utf8("Timestamp(Millisecond, None)")),IntervalMonthDayNano("92233720368547758080")) | -+-----------------------------------------------------------------------------------------------------------------------------------------+ -| 2023-12-01T07:39:46.222 | -+-----------------------------------------------------------------------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| date_sub(arrow_cast(Utf8("2023-12-06 07:39:46.222"),Utf8("Timestamp(Millisecond, None)")),IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 5, nanoseconds: 0 }")) | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-12-01T07:39:46.222 | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT date_sub('2023-12-06 07:39:46.222'::TIMESTAMP_MS, '5 day'); @@ -50,11 +50,11 @@ SELECT date_sub('2023-12-06 07:39:46.222'::TIMESTAMP_MS, '5 day'); SELECT date_sub('2023-12-06'::DATE, INTERVAL '3 month 5 day'); -+-------------------------------------------------------------------------------------+ -| date_sub(Utf8("2023-12-06"),IntervalMonthDayNano("237684487635026733149179609088")) | -+-------------------------------------------------------------------------------------+ -| 2023-09-01 | -+-------------------------------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------------------+ +| date_sub(Utf8("2023-12-06"),IntervalMonthDayNano("IntervalMonthDayNano { months: 3, days: 5, nanoseconds: 0 }")) | ++------------------------------------------------------------------------------------------------------------------+ +| 2023-09-01 | ++------------------------------------------------------------------------------------------------------------------+ SELECT date_sub('2023-12-06'::DATE, '3 month 5 day'); @@ -114,13 +114,13 @@ Affected Rows: 1 --- date_add --- SELECT date_add(d, INTERVAL '1 year 2 month 3 day') from dates; -+---------------------------------------------------------------------------+ -| date_add(dates.d,IntervalMonthDayNano("1109194275255040958530743959552")) | -+---------------------------------------------------------------------------+ -| 1993-03-04 | -| 1995-03-03 | -| 2025-02-09 | -+---------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------+ +| date_add(dates.d,IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 0 }")) | ++--------------------------------------------------------------------------------------------------------+ +| 1993-03-04 | +| 1995-03-03 | +| 2025-02-09 | ++--------------------------------------------------------------------------------------------------------+ SELECT date_add(d, '1 year 2 month 3 day') from dates; @@ -134,13 +134,13 @@ SELECT date_add(d, '1 year 2 month 3 day') from dates; SELECT date_add(ts, INTERVAL '1 year 2 month 3 day') from dates; -+----------------------------------------------------------------------------+ -| date_add(dates.ts,IntervalMonthDayNano("1109194275255040958530743959552")) | -+----------------------------------------------------------------------------+ -| 1971-03-04T00:00:00.001 | -| 1971-03-04T00:00:00.002 | -| 1971-03-04T00:00:00.003 | -+----------------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------------+ +| date_add(dates.ts,IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 0 }")) | ++---------------------------------------------------------------------------------------------------------+ +| 1971-03-04T00:00:00.001 | +| 1971-03-04T00:00:00.002 | +| 1971-03-04T00:00:00.003 | ++---------------------------------------------------------------------------------------------------------+ SELECT date_add(ts, '1 year 2 month 3 day') from dates; @@ -155,13 +155,13 @@ SELECT date_add(ts, '1 year 2 month 3 day') from dates; --- date_sub --- SELECT date_sub(d, INTERVAL '1 year 2 month 3 day') from dates; -+---------------------------------------------------------------------------+ -| date_sub(dates.d,IntervalMonthDayNano("1109194275255040958530743959552")) | -+---------------------------------------------------------------------------+ -| 1990-10-29 | -| 1992-10-27 | -| 2022-10-03 | -+---------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------+ +| date_sub(dates.d,IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 0 }")) | ++--------------------------------------------------------------------------------------------------------+ +| 1990-10-29 | +| 1992-10-27 | +| 2022-10-03 | ++--------------------------------------------------------------------------------------------------------+ SELECT date_sub(d, '1 year 2 month 3 day') from dates; @@ -175,13 +175,13 @@ SELECT date_sub(d, '1 year 2 month 3 day') from dates; SELECT date_sub(ts, INTERVAL '1 year 2 month 3 day') from dates; -+----------------------------------------------------------------------------+ -| date_sub(dates.ts,IntervalMonthDayNano("1109194275255040958530743959552")) | -+----------------------------------------------------------------------------+ -| 1968-10-29T00:00:00.001 | -| 1968-10-29T00:00:00.002 | -| 1968-10-29T00:00:00.003 | -+----------------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------------+ +| date_sub(dates.ts,IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 0 }")) | ++---------------------------------------------------------------------------------------------------------+ +| 1968-10-29T00:00:00.001 | +| 1968-10-29T00:00:00.002 | +| 1968-10-29T00:00:00.003 | ++---------------------------------------------------------------------------------------------------------+ SELECT date_sub(ts, '1 year 2 month 3 day') from dates; diff --git a/tests/cases/standalone/common/function/geo.result b/tests/cases/standalone/common/function/geo.result index 4255a7db35..e251cdd0a1 100644 --- a/tests/cases/standalone/common/function/geo.result +++ b/tests/cases/standalone/common/function/geo.result @@ -32,11 +32,11 @@ Error: 3001(EngineExecuteQuery), H3 error: invalid resolution (got Some(255)): o SELECT h3_latlng_to_cell(37.76938, -122.3889, 8::Int8), h3_latlng_to_cell_string(37.76938, -122.3889, 8::Int8); -+------------------------------------------------------------------+-------------------------------------------------------------------------+ -| h3_latlng_to_cell(Float64(37.76938),Float64(-122.3889),Int64(8)) | h3_latlng_to_cell_string(Float64(37.76938),Float64(-122.3889),Int64(8)) | -+------------------------------------------------------------------+-------------------------------------------------------------------------+ -| 613196570438926335 | 88283082e7fffff | -+------------------------------------------------------------------+-------------------------------------------------------------------------+ ++-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------+ +| h3_latlng_to_cell(Float64(37.76938),Float64(-122.3889),arrow_cast(Int64(8),Utf8("Int8"))) | h3_latlng_to_cell_string(Float64(37.76938),Float64(-122.3889),arrow_cast(Int64(8),Utf8("Int8"))) | ++-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------+ +| 613196570438926335 | 88283082e7fffff | ++-------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------+ SELECT h3_latlng_to_cell(37.76938, -122.3889, 8::Int16), h3_latlng_to_cell_string(37.76938, -122.3889, 8::Int16); @@ -224,11 +224,11 @@ Error: 3001(EngineExecuteQuery), Invalid geohash resolution -1, expect value: [1 SELECT geohash(37.76938, -122.3889, 11::Int8); -+---------------------------------------------------------+ -| geohash(Float64(37.76938),Float64(-122.3889),Int64(11)) | -+---------------------------------------------------------+ -| 9q8yygxneft | -+---------------------------------------------------------+ ++----------------------------------------------------------------------------------+ +| geohash(Float64(37.76938),Float64(-122.3889),arrow_cast(Int64(11),Utf8("Int8"))) | ++----------------------------------------------------------------------------------+ +| 9q8yygxneft | ++----------------------------------------------------------------------------------+ SELECT geohash(37.76938, -122.3889, 11::Int16); diff --git a/tests/cases/standalone/common/information_schema/region_peers.result b/tests/cases/standalone/common/information_schema/region_peers.result index 0c7919e5dd..af58215148 100644 --- a/tests/cases/standalone/common/information_schema/region_peers.result +++ b/tests/cases/standalone/common/information_schema/region_peers.result @@ -42,7 +42,7 @@ Affected Rows: 0 SELECT COUNT(distinct region_id) FROM region_peers; +----------------------------------------+ -| COUNT(DISTINCT region_peers.region_id) | +| count(DISTINCT region_peers.region_id) | +----------------------------------------+ | 6 | +----------------------------------------+ diff --git a/tests/cases/standalone/common/information_schema/region_statistics.result b/tests/cases/standalone/common/information_schema/region_statistics.result index 9c7addf316..aa8e649ec2 100644 --- a/tests/cases/standalone/common/information_schema/region_statistics.result +++ b/tests/cases/standalone/common/information_schema/region_statistics.result @@ -27,7 +27,7 @@ SELECT SUM(region_rows), SUM(disk_size), SUM(sst_size), SUM(index_size) IN (SELECT TABLE_ID FROM INFORMATION_SCHEMA.TABLES WHERE table_name = 'test' and table_schema = 'public'); +-------------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+------------------------------------------------------+ -| SUM(information_schema.region_statistics.region_rows) | SUM(information_schema.region_statistics.disk_size) | SUM(information_schema.region_statistics.sst_size) | SUM(information_schema.region_statistics.index_size) | +| sum(information_schema.region_statistics.region_rows) | sum(information_schema.region_statistics.disk_size) | sum(information_schema.region_statistics.sst_size) | sum(information_schema.region_statistics.index_size) | +-------------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+------------------------------------------------------+ | 3 | 2238 | 0 | 0 | +-------------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+------------------------------------------------------+ diff --git a/tests/cases/standalone/common/insert/big_insert.result b/tests/cases/standalone/common/insert/big_insert.result index 0506b912b9..18a90a5d44 100644 --- a/tests/cases/standalone/common/insert/big_insert.result +++ b/tests/cases/standalone/common/insert/big_insert.result @@ -9,7 +9,7 @@ Affected Rows: 1050 SELECT COUNT(*) FROM integers; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 10 | +----------+ diff --git a/tests/cases/standalone/common/order/limit.result b/tests/cases/standalone/common/order/limit.result index dd9dec772f..999a3a1087 100644 --- a/tests/cases/standalone/common/order/limit.result +++ b/tests/cases/standalone/common/order/limit.result @@ -16,7 +16,7 @@ SELECT a FROM test LIMIT 1; SELECT a FROM test LIMIT 1.25; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected LIMIT to be an integer or null, but got Float64 SELECT a FROM test LIMIT 2-1; @@ -28,19 +28,19 @@ SELECT a FROM test LIMIT 2-1; SELECT a FROM test LIMIT a; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3000(PlanQuery), Failed to plan SQL: No field named a. SELECT a FROM test LIMIT a+1; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3000(PlanQuery), Failed to plan SQL: No field named a. SELECT a FROM test LIMIT SUM(42); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: This feature is not implemented: Unsupported LIMIT expression: Some(AggregateFunction(AggregateFunction { func: AggregateUDF { inner: Sum { signature: Signature { type_signature: UserDefined, volatility: Immutable } } }, args: [Literal(Int64(42))], distinct: false, filter: None, order_by: None, null_treatment: None })) SELECT a FROM test LIMIT row_number() OVER (); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: This feature is not implemented: Unsupported LIMIT expression: Some(Cast(Cast { expr: WindowFunction(WindowFunction { fun: WindowUDF(WindowUDF { inner: RowNumber { signature: Signature { type_signature: NullAry, volatility: Immutable } } }), args: [], partition_by: [], order_by: [], window_frame: WindowFrame { units: Rows, start_bound: Preceding(UInt64(NULL)), end_bound: Following(UInt64(NULL)), is_causal: false }, null_treatment: None }), data_type: Int64 })) CREATE TABLE test2 (a STRING, ts TIMESTAMP TIME INDEX); @@ -60,7 +60,7 @@ SELECT * FROM test2 LIMIT 3; select 1 limit date '1992-01-01'; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected LIMIT to be an integer or null, but got Date32 CREATE TABLE integers(i TIMESTAMP TIME INDEX); @@ -93,35 +93,35 @@ SELECT * FROM integers LIMIT 4; SELECT * FROM integers as int LIMIT (SELECT MIN(integers.i) FROM integers); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected LIMIT to be an integer or null, but got Timestamp(Millisecond, None) SELECT * FROM integers as int OFFSET (SELECT MIN(integers.i) FROM integers); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in OFFSET clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected OFFSET to be an integer or null, but got Timestamp(Millisecond, None) SELECT * FROM integers as int LIMIT (SELECT MAX(integers.i) FROM integers) OFFSET (SELECT MIN(integers.i) FROM integers); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in OFFSET clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected LIMIT to be an integer or null, but got Timestamp(Millisecond, None) SELECT * FROM integers as int LIMIT (SELECT max(integers.i) FROM integers where i > 5); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Cannot infer common argument type for comparison operation Timestamp(Millisecond, None) > Int64 SELECT * FROM integers as int LIMIT (SELECT max(integers.i) FROM integers where i > 5); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Cannot infer common argument type for comparison operation Timestamp(Millisecond, None) > Int64 SELECT * FROM integers as int LIMIT (SELECT NULL); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: This feature is not implemented: Unsupported LIMIT expression: Some(ScalarSubquery()) SELECT * FROM integers as int LIMIT (SELECT -1); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: This feature is not implemented: Unsupported LIMIT expression: Some(ScalarSubquery()) SELECT * FROM integers as int LIMIT (SELECT 'ab'); -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Unexpected expression in LIMIT clause +Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Expected LIMIT to be an integer or null, but got Utf8 DROP TABLE integers; diff --git a/tests/cases/standalone/common/order/limit_union.result b/tests/cases/standalone/common/order/limit_union.result index a6f86768ea..d7dcf11c02 100644 --- a/tests/cases/standalone/common/order/limit_union.result +++ b/tests/cases/standalone/common/order/limit_union.result @@ -23,7 +23,7 @@ SELECT * FROM integers UNION ALL SELECT * FROM integers LIMIT 7; SELECT COUNT(*) FROM (SELECT * FROM integers UNION ALL SELECT * FROM integers LIMIT 7) tbl; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 7 | +----------+ diff --git a/tests/cases/standalone/common/order/nulls_first.result b/tests/cases/standalone/common/order/nulls_first.result index 8aae8ccadb..26b9c0c156 100644 --- a/tests/cases/standalone/common/order/nulls_first.result +++ b/tests/cases/standalone/common/order/nulls_first.result @@ -41,7 +41,7 @@ SELECT * FROM test ORDER BY i NULLS LAST, j NULLS FIRST; SELECT i, j, row_number() OVER (PARTITION BY i ORDER BY j NULLS LAST) FROM test ORDER BY i NULLS FIRST, j NULLS FIRST; +---+---+-----------------------------------------------------------------------------------------------------------------------+ -| i | j | ROW_NUMBER() PARTITION BY [test.i] ORDER BY [test.j ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| i | j | row_number() PARTITION BY [test.i] ORDER BY [test.j ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +---+---+-----------------------------------------------------------------------------------------------------------------------+ | | 1 | 1 | | 1 | | 2 | diff --git a/tests/cases/standalone/common/order/order_by.result b/tests/cases/standalone/common/order/order_by.result index 8cf92d9fbf..eba412dba0 100644 --- a/tests/cases/standalone/common/order/order_by.result +++ b/tests/cases/standalone/common/order/order_by.result @@ -148,7 +148,7 @@ SELECT b, a FROM test WHERE a < 13 ORDER BY b DESC; SELECT b % 2 AS f, SUM(a) FROM test GROUP BY f ORDER BY b % 2; +---+-------------+ -| f | SUM(test.a) | +| f | sum(test.a) | +---+-------------+ | 0 | 24 | | 1 | 12 | @@ -167,7 +167,7 @@ SELECT b % 2 AS f, a FROM test ORDER BY b % 2, a; SELECT b % 2 AS f, SUM(a) FROM test GROUP BY f ORDER BY f; +---+-------------+ -| f | SUM(test.a) | +| f | sum(test.a) | +---+-------------+ | 0 | 24 | | 1 | 12 | @@ -176,7 +176,7 @@ SELECT b % 2 AS f, SUM(a) FROM test GROUP BY f ORDER BY f; SELECT b % 2 AS f, SUM(a) FROM test GROUP BY f ORDER BY 1; +---+-------------+ -| f | SUM(test.a) | +| f | sum(test.a) | +---+-------------+ | 0 | 24 | | 1 | 12 | @@ -292,26 +292,21 @@ explain analyze select tag from t where num > 6 order by ts desc limit 2; +-+-+-+ | stage | node | plan_| +-+-+-+ -| 0_| 0_|_GlobalLimitExec: skip=0, fetch=2 REDACTED -|_|_|_SortPreservingMergeExec: [ts@1 DESC] REDACTED -|_|_|_SortExec: expr=[ts@1 DESC], preserve_partitioning=[true] REDACTED +| 0_| 0_|_SortPreservingMergeExec: [ts@1 DESC], fetch=2 REDACTED +|_|_|_SortExec: TopK(fetch=2), expr=[ts@1 DESC], preserve_partitioning=[true] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_GlobalLimitExec: skip=0, fetch=2 REDACTED -|_|_|_SortPreservingMergeExec: [ts@1 DESC] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [ts@1 DESC], fetch=2 REDACTED |_|_|_SortExec: TopK(fetch=2), expr=[ts@1 DESC], preserve_partitioning=[true] REDACTED -|_|_|_ProjectionExec: expr=[tag@0 as tag, ts@1 as ts] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_FilterExec: num@2 > 6 REDACTED +|_|_|_FilterExec: num@2 > 6, projection=[tag@0, ts@1] REDACTED |_|_|_RepartitionExec: partitioning=REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=1 (1 memtable ranges, 0 file 0 ranges) REDACTED |_|_|_| -| 1_| 1_|_GlobalLimitExec: skip=0, fetch=2 REDACTED -|_|_|_SortPreservingMergeExec: [ts@1 DESC] REDACTED +| 1_| 1_|_SortPreservingMergeExec: [ts@1 DESC], fetch=2 REDACTED |_|_|_SortExec: TopK(fetch=2), expr=[ts@1 DESC], preserve_partitioning=[true] REDACTED -|_|_|_ProjectionExec: expr=[tag@0 as tag, ts@1 as ts] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_FilterExec: num@2 > 6 REDACTED +|_|_|_FilterExec: num@2 > 6, projection=[tag@0, ts@1] REDACTED |_|_|_RepartitionExec: partitioning=REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=1 (1 memtable ranges, 0 file 0 ranges) REDACTED |_|_|_| diff --git a/tests/cases/standalone/common/order/order_by_exceptions.result b/tests/cases/standalone/common/order/order_by_exceptions.result index 9679c45958..0a3c47a272 100644 --- a/tests/cases/standalone/common/order/order_by_exceptions.result +++ b/tests/cases/standalone/common/order/order_by_exceptions.result @@ -89,7 +89,7 @@ EXPLAIN SELECT a % 2, b FROM test UNION SELECT a % 2 AS k, b FROM test ORDER BY SELECT a % 2, b FROM test UNION SELECT a % 2 AS k FROM test ORDER BY -1; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Union queries must have the same number of columns, (left is 2, right is 1) +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: UNION queries have different number of columns: left has 2 columns whereas right has 1 columns DROP TABLE test; diff --git a/tests/cases/standalone/common/order/windowed_sort.result b/tests/cases/standalone/common/order/windowed_sort.result index 3ae8f9f846..bf5cabdad8 100644 --- a/tests/cases/standalone/common/order/windowed_sort.result +++ b/tests/cases/standalone/common/order/windowed_sort.result @@ -67,8 +67,7 @@ EXPLAIN ANALYZE SELECT * FROM test ORDER BY t LIMIT 5; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_GlobalLimitExec: skip=0, fetch=5 REDACTED -|_|_|_SortPreservingMergeExec: [t@1 ASC NULLS LAST] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [t@1 ASC NULLS LAST], fetch=5 REDACTED |_|_|_WindowedSortExec: expr=t@1 ASC NULLS LAST num_ranges=4 fetch=5 REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=4 (1 memtable ranges, 3 file 3 ranges) REDACTED |_|_|_| @@ -99,8 +98,7 @@ EXPLAIN ANALYZE SELECT * FROM test ORDER BY t DESC LIMIT 5; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_GlobalLimitExec: skip=0, fetch=5 REDACTED -|_|_|_SortPreservingMergeExec: [t@1 DESC] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [t@1 DESC], fetch=5 REDACTED |_|_|_WindowedSortExec: expr=t@1 DESC num_ranges=4 fetch=5 REDACTED |_|_|_PartSortExec: expr=t@1 DESC num_ranges=4 limit=5 REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=4 (1 memtable ranges, 3 file 3 ranges) REDACTED @@ -181,8 +179,7 @@ EXPLAIN ANALYZE SELECT * FROM test_pk ORDER BY t LIMIT 5; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_GlobalLimitExec: skip=0, fetch=5 REDACTED -|_|_|_SortPreservingMergeExec: [t@2 ASC NULLS LAST] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [t@2 ASC NULLS LAST], fetch=5 REDACTED |_|_|_WindowedSortExec: expr=t@2 ASC NULLS LAST num_ranges=4 fetch=5 REDACTED |_|_|_PartSortExec: expr=t@2 ASC NULLS LAST num_ranges=4 limit=5 REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=4 (1 memtable ranges, 3 file 3 ranges) REDACTED @@ -214,8 +211,7 @@ EXPLAIN ANALYZE SELECT * FROM test_pk ORDER BY t DESC LIMIT 5; +-+-+-+ | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_GlobalLimitExec: skip=0, fetch=5 REDACTED -|_|_|_SortPreservingMergeExec: [t@2 DESC] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [t@2 DESC], fetch=5 REDACTED |_|_|_WindowedSortExec: expr=t@2 DESC num_ranges=4 fetch=5 REDACTED |_|_|_PartSortExec: expr=t@2 DESC num_ranges=4 limit=5 REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=4 (1 memtable ranges, 3 file 3 ranges) REDACTED diff --git a/tests/cases/standalone/common/parser/parser.result b/tests/cases/standalone/common/parser/parser.result index 7e6dce85b7..a2bc95aaed 100644 --- a/tests/cases/standalone/common/parser/parser.result +++ b/tests/cases/standalone/common/parser/parser.result @@ -11,7 +11,7 @@ Affected Rows: 2 SELECT i, j FROM (SELECT COLUMNS(*)::VARCHAR FROM integers); Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Invalid function 'columns'. -Did you mean 'COUNT'? +Did you mean 'count'? SELECT i, j FROM (SELECT * FROM integers); diff --git a/tests/cases/standalone/common/prepare/mysql_prepare.result b/tests/cases/standalone/common/prepare/mysql_prepare.result index ae6ca4400c..31fa6a9c9a 100644 --- a/tests/cases/standalone/common/prepare/mysql_prepare.result +++ b/tests/cases/standalone/common/prepare/mysql_prepare.result @@ -3,7 +3,7 @@ -- SQLNESS PROTOCOL MYSQL SELECT ?; -Failed to execute query, err: MySqlError { ERROR 1815 (HY000): (PlanQuery): Failed to plan SQL: Error during planning: Placeholder type could not be resolved. Make sure that the placeholder is bound to a concrete type, e.g. by providing parameter values. } +Failed to execute query, err: MySqlError { ERROR 1815 (HY000): (EngineExecuteQuery): DataFusion error: Execution error: Placeholder '?' was not provided a value for execution. } -- SQLNESS PROTOCOL MYSQL PREPARE stmt FROM 'SELECT ?::int;'; diff --git a/tests/cases/standalone/common/promql/scalar.result b/tests/cases/standalone/common/promql/scalar.result index 9e56130ce9..7ad536e5b4 100644 --- a/tests/cases/standalone/common/promql/scalar.result +++ b/tests/cases/standalone/common/promql/scalar.result @@ -128,7 +128,7 @@ TQL EVAL (0, 15, '5s') scalar(host{host="host1"}) + host; TQL EVAL (0, 15, '5s') scalar(count(count(host) by (host))); +---------------------+--------------------------------+ -| ts | scalar(COUNT(COUNT(host.val))) | +| ts | scalar(count(count(host.val))) | +---------------------+--------------------------------+ | 1970-01-01T00:00:00 | 2.0 | | 1970-01-01T00:00:05 | 2.0 | diff --git a/tests/cases/standalone/common/promql/simple_histogram.result b/tests/cases/standalone/common/promql/simple_histogram.result index a92e962971..be462cc4bb 100644 --- a/tests/cases/standalone/common/promql/simple_histogram.result +++ b/tests/cases/standalone/common/promql/simple_histogram.result @@ -275,7 +275,7 @@ Affected Rows: 12 tql eval (3000, 3005, '3s') histogram_quantile(0.5, sum by(le, s) (rate(histogram3_bucket[5m]))); +---+---------------------+---------------------------------+ -| s | ts | SUM(prom_rate(ts_range,val,ts)) | +| s | ts | sum(prom_rate(ts_range,val,ts)) | +---+---------------------+---------------------------------+ | a | 1970-01-01T00:50:00 | 0.55 | | a | 1970-01-01T00:50:03 | 0.5500000000000002 | diff --git a/tests/cases/standalone/common/promql/time_fn.result b/tests/cases/standalone/common/promql/time_fn.result index a3a3997e3f..f5ce129a60 100644 --- a/tests/cases/standalone/common/promql/time_fn.result +++ b/tests/cases/standalone/common/promql/time_fn.result @@ -161,8 +161,8 @@ tql eval (1, 2, '1s') hour(); +---------------------+------------------------------+ | time | date_part(Utf8("hour"),time) | +---------------------+------------------------------+ -| 1970-01-01T00:00:01 | 0.0 | -| 1970-01-01T00:00:02 | 0.0 | +| 1970-01-01T00:00:01 | 0 | +| 1970-01-01T00:00:02 | 0 | +---------------------+------------------------------+ tql eval (1, 2, '1s') hour(metrics); @@ -170,8 +170,8 @@ tql eval (1, 2, '1s') hour(metrics); +---------------------+----------------------------+ | ts | date_part(Utf8("hour"),ts) | +---------------------+----------------------------+ -| 1970-01-01T00:00:01 | 0.0 | -| 1970-01-01T00:00:02 | 0.0 | +| 1970-01-01T00:00:01 | 0 | +| 1970-01-01T00:00:02 | 0 | +---------------------+----------------------------+ -- 2023-12-01T06:43:43Z @@ -180,7 +180,7 @@ tql eval (1701413023, 1701413023, '1s') hour(); +---------------------+------------------------------+ | time | date_part(Utf8("hour"),time) | +---------------------+------------------------------+ -| 2023-12-01T06:43:43 | 6.0 | +| 2023-12-01T06:43:43 | 6 | +---------------------+------------------------------+ tql eval (1701413023, 1701413023, '1s') hour(metrics); @@ -193,7 +193,7 @@ tql eval (1701413023, 1701413023, '1s') minute(); +---------------------+--------------------------------+ | time | date_part(Utf8("minute"),time) | +---------------------+--------------------------------+ -| 2023-12-01T06:43:43 | 43.0 | +| 2023-12-01T06:43:43 | 43 | +---------------------+--------------------------------+ tql eval (1701413023, 1701413023, '1s') month(); @@ -201,7 +201,7 @@ tql eval (1701413023, 1701413023, '1s') month(); +---------------------+-------------------------------+ | time | date_part(Utf8("month"),time) | +---------------------+-------------------------------+ -| 2023-12-01T06:43:43 | 12.0 | +| 2023-12-01T06:43:43 | 12 | +---------------------+-------------------------------+ tql eval (1701413023, 1701413023, '1s') year(); @@ -209,7 +209,7 @@ tql eval (1701413023, 1701413023, '1s') year(); +---------------------+------------------------------+ | time | date_part(Utf8("year"),time) | +---------------------+------------------------------+ -| 2023-12-01T06:43:43 | 2023.0 | +| 2023-12-01T06:43:43 | 2023 | +---------------------+------------------------------+ tql eval (1701413023, 1701413023, '1s') day_of_month(); @@ -217,7 +217,7 @@ tql eval (1701413023, 1701413023, '1s') day_of_month(); +---------------------+-----------------------------+ | time | date_part(Utf8("day"),time) | +---------------------+-----------------------------+ -| 2023-12-01T06:43:43 | 1.0 | +| 2023-12-01T06:43:43 | 1 | +---------------------+-----------------------------+ tql eval (1701413023, 1701413023, '1s') day_of_week(); @@ -225,7 +225,7 @@ tql eval (1701413023, 1701413023, '1s') day_of_week(); +---------------------+-----------------------------+ | time | date_part(Utf8("dow"),time) | +---------------------+-----------------------------+ -| 2023-12-01T06:43:43 | 5.0 | +| 2023-12-01T06:43:43 | 5 | +---------------------+-----------------------------+ tql eval (1701413023, 1701413023, '1s') day_of_year(); @@ -233,7 +233,7 @@ tql eval (1701413023, 1701413023, '1s') day_of_year(); +---------------------+-----------------------------+ | time | date_part(Utf8("doy"),time) | +---------------------+-----------------------------+ -| 2023-12-01T06:43:43 | 335.0 | +| 2023-12-01T06:43:43 | 335 | +---------------------+-----------------------------+ -- 2024-01-01T06:43:43Z leap year @@ -242,125 +242,125 @@ tql eval (1704091423, 1704091423, '1s') day_of_year(); +---------------------+-----------------------------+ | time | date_part(Utf8("doy"),time) | +---------------------+-----------------------------+ -| 2024-01-01T06:43:43 | 1.0 | +| 2024-01-01T06:43:43 | 1 | +---------------------+-----------------------------+ -- 2023-01-01T06:43:43Z tql eval (1672555423, 1672555423, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-01-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-01-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-02-01T06:43:43Z tql eval (1675233823, 1675233823, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-02-01T06:43:43 | 28.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-02-01T06:43:43 | 28 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2024-02-01T06:43:43Z leap year tql eval (1706769823, 1706769823, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2024-02-01T06:43:43 | 29.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2024-02-01T06:43:43 | 29 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-03-01T06:43:43Z tql eval (1677653023, 1677653023, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-03-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-03-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-04-01T06:43:43Z tql eval (1680331423, 1680331423, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-04-01T06:43:43 | 30.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-04-01T06:43:43 | 30 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-05-01T06:43:43Z tql eval (1682923423, 1682923423, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-05-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-05-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-06-01T06:43:43Z tql eval (1685601823, 1685601823, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-06-01T06:43:43 | 30.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-06-01T06:43:43 | 30 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-07-01T06:43:43Z tql eval (1688193823, 1688193823, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-07-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-07-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-08-01T06:43:43Z tql eval (1690872223, 1690872223, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-08-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-08-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-09-01T06:43:43Z tql eval (1693550623, 1693550623, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-09-01T06:43:43 | 30.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-09-01T06:43:43 | 30 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-10-01T06:43:43Z tql eval (1696142623, 1696142623, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-10-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-10-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-11-01T06:43:43Z tql eval (1698821023, 1698821023, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-11-01T06:43:43 | 30.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-11-01T06:43:43 | 30 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ -- 2023-12-01T06:43:43Z tql eval (1701413023, 1701413023, '1s') days_in_month(); -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("4294967296")) | -+---------------------+----------------------------------------------------------------------------------------------------------------+ -| 2023-12-01T06:43:43 | 31.0 | -+---------------------+----------------------------------------------------------------------------------------------------------------+ ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| time | date_part(Utf8("day"),date_trunc(Utf8("month"),time) + IntervalYearMonth("1") - IntervalDayTime("IntervalDayTime { days: 1, milliseconds: 0 }")) | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2023-12-01T06:43:43 | 31 | ++---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+ drop table metrics; diff --git a/tests/cases/standalone/common/range/by.result b/tests/cases/standalone/common/range/by.result index e81822e1b8..a31e4d42e8 100644 --- a/tests/cases/standalone/common/range/by.result +++ b/tests/cases/standalone/common/range/by.result @@ -24,7 +24,7 @@ Affected Rows: 10 SELECT ts, length(host), max(val) RANGE '5s' FROM host ALIGN '20s' BY (length(host)) ORDER BY ts; +---------------------+-----------------------------+------------------------+ -| ts | character_length(host.host) | MAX(host.val) RANGE 5s | +| ts | character_length(host.host) | max(host.val) RANGE 5s | +---------------------+-----------------------------+------------------------+ | 1970-01-01T00:00:00 | 5 | 3 | | 1970-01-01T00:00:20 | 5 | 5 | @@ -33,7 +33,7 @@ SELECT ts, length(host), max(val) RANGE '5s' FROM host ALIGN '20s' BY (length(ho SELECT ts, max(val) RANGE '5s' FROM host ALIGN '20s' BY (2) ORDER BY ts; +---------------------+------------------------+ -| ts | MAX(host.val) RANGE 5s | +| ts | max(host.val) RANGE 5s | +---------------------+------------------------+ | 1970-01-01T00:00:00 | 3 | | 1970-01-01T00:00:20 | 5 | @@ -44,7 +44,7 @@ SELECT ts, max(val) RANGE '5s' FROM host ALIGN '20s' BY (2) ORDER BY ts; SELECT ts, max(val) RANGE '5s' FROM host ALIGN '20s' BY () ORDER BY ts; +---------------------+------------------------+ -| ts | MAX(host.val) RANGE 5s | +| ts | max(host.val) RANGE 5s | +---------------------+------------------------+ | 1970-01-01T00:00:00 | 3 | | 1970-01-01T00:00:20 | 5 | @@ -53,7 +53,7 @@ SELECT ts, max(val) RANGE '5s' FROM host ALIGN '20s' BY () ORDER BY ts; SELECT ts, length(host)::INT64 + 2, max(val) RANGE '5s' FROM host ALIGN '20s' BY (length(host)::INT64 + 2) ORDER BY ts; +---------------------+------------------------------------------------------------------+------------------------+ -| ts | arrow_cast(character_length(host.host),Utf8("Int64")) + Int64(2) | MAX(host.val) RANGE 5s | +| ts | arrow_cast(character_length(host.host),Utf8("Int64")) + Int64(2) | max(host.val) RANGE 5s | +---------------------+------------------------------------------------------------------+------------------------+ | 1970-01-01T00:00:00 | 7 | 3 | | 1970-01-01T00:00:20 | 7 | 5 | @@ -63,7 +63,7 @@ SELECT ts, length(host)::INT64 + 2, max(val) RANGE '5s' FROM host ALIGN '20s' BY -- project non-aggregation key SELECT ts, host, max(val) RANGE '5s' FROM host ALIGN '20s' BY () ORDER BY ts; -Error: 3001(EngineExecuteQuery), DataFusion error: No field named host.host. Valid fields are "MAX(host.val) RANGE 5s", host.ts, "Int64(1)". +Error: 3001(EngineExecuteQuery), DataFusion error: No field named host.host. Valid fields are "max(host.val) RANGE 5s", host.ts, "Int64(1)". DROP TABLE host; @@ -95,7 +95,7 @@ Affected Rows: 10 SELECT ts, max(val) RANGE '5s' FROM host ALIGN '20s' ORDER BY ts; +---------------------+------------------------+ -| ts | MAX(host.val) RANGE 5s | +| ts | max(host.val) RANGE 5s | +---------------------+------------------------+ | 1970-01-01T00:00:00 | 3 | | 1970-01-01T00:00:20 | 5 | diff --git a/tests/cases/standalone/common/range/calculate.result b/tests/cases/standalone/common/range/calculate.result index 50030a29ee..416ee2d283 100644 --- a/tests/cases/standalone/common/range/calculate.result +++ b/tests/cases/standalone/common/range/calculate.result @@ -39,7 +39,7 @@ SELECT ts, host, covar(val::DOUBLE, val::DOUBLE) RANGE '20s' FROM host ALIGN '10 SELECT ts, host, 2 * min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-----------------------------------+ -| ts | host | Int64(2) * MIN(host.val) RANGE 5s | +| ts | host | Int64(2) * min(host.val) RANGE 5s | +---------------------+-------+-----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -56,7 +56,7 @@ SELECT ts, host, 2 * min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; SELECT ts, host, min(val * 2) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-----------------------------------+ -| ts | host | MIN(host.val * Int64(2)) RANGE 5s | +| ts | host | min(host.val * Int64(2)) RANGE 5s | +---------------------+-------+-----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -73,7 +73,7 @@ SELECT ts, host, min(val * 2) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; SELECT ts, host, min(val::DOUBLE) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 5s | +| ts | host | min(host.val) RANGE 5s | +---------------------+-------+------------------------+ | 1970-01-01T00:00:00 | host1 | 0.0 | | 1970-01-01T00:00:05 | host1 | | @@ -90,7 +90,7 @@ SELECT ts, host, min(val::DOUBLE) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, SELECT ts, host, min(floor(val::DOUBLE)) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-------------------------------+ -| ts | host | MIN(floor(host.val)) RANGE 5s | +| ts | host | min(floor(host.val)) RANGE 5s | +---------------------+-------+-------------------------------+ | 1970-01-01T00:00:00 | host1 | 0.0 | | 1970-01-01T00:00:05 | host1 | | @@ -107,7 +107,7 @@ SELECT ts, host, min(floor(val::DOUBLE)) RANGE '5s' FROM host ALIGN '5s' ORDER B SELECT ts, host, floor(min(val) RANGE '5s') FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-------------------------------+ -| ts | host | floor(MIN(host.val) RANGE 5s) | +| ts | host | floor(min(host.val) RANGE 5s) | +---------------------+-------+-------------------------------+ | 1970-01-01T00:00:00 | host1 | 0.0 | | 1970-01-01T00:00:05 | host1 | | @@ -125,7 +125,7 @@ SELECT ts, host, floor(min(val) RANGE '5s') FROM host ALIGN '5s' ORDER BY host, SELECT ts, host, (min(val) + max(val)) RANGE '20s' + 1.0 FROM host ALIGN '10s' ORDER BY host, ts; +---------------------+-------+----------------------------------------------------------------+ -| ts | host | MIN(host.val) RANGE 20s + MAX(host.val) RANGE 20s + Float64(1) | +| ts | host | min(host.val) RANGE 20s + max(host.val) RANGE 20s + Float64(1) | +---------------------+-------+----------------------------------------------------------------+ | 1969-12-31T23:59:50 | host1 | 1.0 | | 1970-01-01T00:00:00 | host1 | 2.0 | @@ -155,7 +155,7 @@ SELECT ts, host, covar(ceil(val::DOUBLE), floor(val::DOUBLE)) RANGE '20s' FROM h SELECT ts, host, floor(cos(ceil(sin(min(val) RANGE '5s')))) FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-----------------------------------------------+ -| ts | host | floor(cos(ceil(sin(MIN(host.val) RANGE 5s)))) | +| ts | host | floor(cos(ceil(sin(min(host.val) RANGE 5s)))) | +---------------------+-------+-----------------------------------------------+ | 1970-01-01T00:00:00 | host1 | 1.0 | | 1970-01-01T00:00:05 | host1 | | @@ -172,7 +172,7 @@ SELECT ts, host, floor(cos(ceil(sin(min(val) RANGE '5s')))) FROM host ALIGN '5s' SELECT ts, host, gcd(CAST(max(floor(val::DOUBLE)) RANGE '10s' FILL PREV as INT64) * 4, max(val * 4) RANGE '10s' FILL PREV) * length(host) + 1 FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| ts | host | gcd(arrow_cast(MAX(floor(host.val)) RANGE 10s FILL PREV,Utf8("Int64")) * Int64(4),MAX(host.val * Int64(4)) RANGE 10s FILL PREV) * character_length(host.host) + Int64(1) | +| ts | host | gcd(arrow_cast(max(floor(host.val)) RANGE 10s FILL PREV,Utf8("Int64")) * Int64(4),max(host.val * Int64(4)) RANGE 10s FILL PREV) * character_length(host.host) + Int64(1) | +---------------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1969-12-31T23:59:55 | host1 | 1 | | 1970-01-01T00:00:00 | host1 | 1 | diff --git a/tests/cases/standalone/common/range/error.result b/tests/cases/standalone/common/range/error.result index 020a9a7d4c..11bcfef6de 100644 --- a/tests/cases/standalone/common/range/error.result +++ b/tests/cases/standalone/common/range/error.result @@ -24,32 +24,32 @@ Affected Rows: 10 -- 1. error timestamp SELECT min(val) RANGE 'not_time' FROM host ALIGN '5s'; -Error: 2000(InvalidSyntax), sql parser error: not a valid duration string: not_time +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: not a valid duration string: not_time SELECT min(val) RANGE '5s' FROM host ALIGN 'not_time'; -Error: 2000(InvalidSyntax), sql parser error: not a valid duration string: not_time +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: not a valid duration string: not_time -- 2.1 no range param SELECT min(val) FROM host ALIGN '5s'; -Error: 2000(InvalidSyntax), sql parser error: Illegal Range select, no RANGE keyword found in any SelectItem +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Illegal Range select, no RANGE keyword found in any SelectItem SELECT 1 FROM host ALIGN '5s'; -Error: 2000(InvalidSyntax), sql parser error: Illegal Range select, no RANGE keyword found in any SelectItem +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Illegal Range select, no RANGE keyword found in any SelectItem SELECT min(val) RANGE '10s', max(val) FROM host ALIGN '5s'; -Error: 3001(EngineExecuteQuery), DataFusion error: No field named "MAX(host.val)". Valid fields are "MIN(host.val) RANGE 10s", host.ts, host.host. +Error: 3001(EngineExecuteQuery), DataFusion error: No field named "max(host.val)". Valid fields are "min(host.val) RANGE 10s", host.ts, host.host. SELECT min(val) * 2 RANGE '10s' FROM host ALIGN '5s'; -Error: 2000(InvalidSyntax), sql parser error: Can't use the RANGE keyword in Expr 2 without function +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Can't use the RANGE keyword in Expr 2 without function SELECT 1 RANGE '10s' FILL NULL FROM host ALIGN '1h' FILL NULL; -Error: 2000(InvalidSyntax), sql parser error: Can't use the RANGE keyword in Expr 1 without function +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Can't use the RANGE keyword in Expr 1 without function -- 2.2 no align param SELECT min(val) RANGE '5s' FROM host; @@ -98,11 +98,11 @@ Error: 3000(PlanQuery), DataFusion error: Error during planning: duration must b SELECT min(val) RANGE '5s' FROM host ALIGN (INTERVAL '0' day); -Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("0")` in range select query +Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 0 }")` in range select query SELECT min(val) RANGE (INTERVAL '0' day) FROM host ALIGN '5s'; -Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("0")` in range select query +Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 0 }")` in range select query DROP TABLE host; diff --git a/tests/cases/standalone/common/range/fill.result b/tests/cases/standalone/common/range/fill.result index 739aec6dd6..28c5c6229e 100644 --- a/tests/cases/standalone/common/range/fill.result +++ b/tests/cases/standalone/common/range/fill.result @@ -24,7 +24,7 @@ Affected Rows: 10 SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 5s | +| ts | host | min(host.val) RANGE 5s | +---------------------+-------+------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -41,7 +41,7 @@ SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' FILL NULL ORDER BY host, ts; +---------------------+-------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL NULL | +| ts | host | min(host.val) RANGE 5s FILL NULL | +---------------------+-------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -58,7 +58,7 @@ SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' FILL NULL ORDER BY hos SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL 6 FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+-------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL 6 | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL 6 | +---------------------+-------+------------------------+-------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0 | | 1970-01-01T00:00:05 | host1 | | 6 | @@ -75,7 +75,7 @@ SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL 6 FROM host ALIGN SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL PREV FROM host ALIGN '5s'ORDER BY host, ts; +---------------------+-------+------------------------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL PREV | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL PREV | +---------------------+-------+------------------------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0 | | 1970-01-01T00:00:05 | host1 | | 0 | @@ -92,7 +92,7 @@ SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL PREV FROM host AL SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL LINEAR FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+------------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL LINEAR | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL LINEAR | +---------------------+-------+------------------------+------------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0.0 | | 1970-01-01T00:00:05 | host1 | | 0.5 | @@ -140,7 +140,7 @@ Affected Rows: 12 SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 5s | +| ts | host | min(host.val) RANGE 5s | +---------------------+-------+------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:15 | host1 | 6 | @@ -151,7 +151,7 @@ SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; SELECT ts, host, min(val) RANGE '5s' FILL NULL FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL NULL | +| ts | host | min(host.val) RANGE 5s FILL NULL | +---------------------+-------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -166,7 +166,7 @@ SELECT ts, host, min(val) RANGE '5s' FILL NULL FROM host ALIGN '5s' ORDER BY hos SELECT ts, host, min(val) RANGE '5s' FILL PREV FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL PREV | +| ts | host | min(host.val) RANGE 5s FILL PREV | +---------------------+-------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | 0 | @@ -181,7 +181,7 @@ SELECT ts, host, min(val) RANGE '5s' FILL PREV FROM host ALIGN '5s' ORDER BY hos SELECT ts, host, min(val) RANGE '5s' FILL LINEAR FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL LINEAR | +| ts | host | min(host.val) RANGE 5s FILL LINEAR | +---------------------+-------+------------------------------------+ | 1970-01-01T00:00:00 | host1 | 0.0 | | 1970-01-01T00:00:05 | host1 | 2.0 | @@ -196,7 +196,7 @@ SELECT ts, host, min(val) RANGE '5s' FILL LINEAR FROM host ALIGN '5s' ORDER BY h SELECT ts, host, min(val) RANGE '5s' FILL 6 FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL 6 | +| ts | host | min(host.val) RANGE 5s FILL 6 | +---------------------+-------+-------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | 6 | @@ -211,7 +211,7 @@ SELECT ts, host, min(val) RANGE '5s' FILL 6 FROM host ALIGN '5s' ORDER BY host, SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' FILL NULL ORDER BY host, ts; +---------------------+-------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s FILL NULL | +| ts | host | min(host.val) RANGE 5s FILL NULL | +---------------------+-------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -226,7 +226,7 @@ SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s' FILL NULL ORDER BY hos SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL NULL FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL NULL | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL NULL | +---------------------+-------+------------------------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0 | | 1970-01-01T00:00:05 | host1 | | | @@ -241,7 +241,7 @@ SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL NULL FROM host AL SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL 6 FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+-------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL 6 | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL 6 | +---------------------+-------+------------------------+-------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0 | | 1970-01-01T00:00:05 | host1 | | 6 | @@ -256,7 +256,7 @@ SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL 6 FROM host ALIGN SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL PREV FROM host ALIGN '5s'ORDER BY host, ts; +---------------------+-------+------------------------+----------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL PREV | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL PREV | +---------------------+-------+------------------------+----------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0 | | 1970-01-01T00:00:05 | host1 | | 0 | @@ -271,7 +271,7 @@ SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL PREV FROM host AL SELECT ts, host, min(val) RANGE '5s', min(val) RANGE '5s' FILL LINEAR FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+------------------------+------------------------------------+ -| ts | host | MIN(host.val) RANGE 5s | MIN(host.val) RANGE 5s FILL LINEAR | +| ts | host | min(host.val) RANGE 5s | min(host.val) RANGE 5s FILL LINEAR | +---------------------+-------+------------------------+------------------------------------+ | 1970-01-01T00:00:00 | host1 | 0 | 0.0 | | 1970-01-01T00:00:05 | host1 | | 2.0 | diff --git a/tests/cases/standalone/common/range/interval.result b/tests/cases/standalone/common/range/interval.result index b67310ab85..6dbb7dd720 100644 --- a/tests/cases/standalone/common/range/interval.result +++ b/tests/cases/standalone/common/range/interval.result @@ -20,18 +20,18 @@ Affected Rows: 8 SELECT ts, host, min(val) RANGE (INTERVAL '1 year') FROM host ALIGN (INTERVAL '1 year') ORDER BY host, ts; -Error: 3000(PlanQuery), DataFusion error: Error during planning: Year or month interval is not allowed in range query: IntervalMonthDayNano("950737950171172051122527404032") +Error: 3000(PlanQuery), DataFusion error: Error during planning: Year or month interval is not allowed in range query: IntervalMonthDayNano("IntervalMonthDayNano { months: 12, days: 0, nanoseconds: 0 }") SELECT ts, host, min(val) RANGE (INTERVAL '1' day) FROM host ALIGN (INTERVAL '1' day) ORDER BY host, ts; -+---------------------+-------+------------------------------------------------------------------+ -| ts | host | MIN(host.val) RANGE IntervalMonthDayNano("18446744073709551616") | -+---------------------+-------+------------------------------------------------------------------+ -| 1970-01-01T00:00:00 | host1 | 0 | -| 1971-01-02T00:00:00 | host1 | 2 | -| 1970-01-01T00:00:00 | host2 | 4 | -| 1971-01-02T00:00:00 | host2 | 6 | -+---------------------+-------+------------------------------------------------------------------+ ++---------------------+-------+---------------------------------------------------------------------------------------------------------+ +| ts | host | min(host.val) RANGE IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") | ++---------------------+-------+---------------------------------------------------------------------------------------------------------+ +| 1970-01-01T00:00:00 | host1 | 0 | +| 1971-01-02T00:00:00 | host1 | 2 | +| 1970-01-01T00:00:00 | host2 | 4 | +| 1971-01-02T00:00:00 | host2 | 6 | ++---------------------+-------+---------------------------------------------------------------------------------------------------------+ DROP TABLE host; diff --git a/tests/cases/standalone/common/range/nest.result b/tests/cases/standalone/common/range/nest.result index 236c0f297a..da4fe15d92 100644 --- a/tests/cases/standalone/common/range/nest.result +++ b/tests/cases/standalone/common/range/nest.result @@ -36,7 +36,7 @@ SELECT ts, host, foo FROM (SELECT ts, host, min(val) RANGE '5s' AS foo FROM host SELECT ts, b, min(c) RANGE '5s' FROM (SELECT ts, host AS b, val AS c FROM host WHERE host = 'host1') ALIGN '5s' BY (b) ORDER BY b, ts; +---------------------+-------+-----------------+ -| ts | b | MIN(c) RANGE 5s | +| ts | b | min(c) RANGE 5s | +---------------------+-------+-----------------+ | 1970-01-01T00:00:00 | host1 | 0 | | 1970-01-01T00:00:05 | host1 | | @@ -55,9 +55,9 @@ EXPLAIN SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s'; +-+-+ | plan_type_| plan_| +-+-+ -| logical_plan_| RangeSelect: range_exprs=[MIN(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host.host], time_index=ts | +| logical_plan_| RangeSelect: range_exprs=[min(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host.host], time_index=ts | |_|_MergeScan [is_placeholder=false]_| -| physical_plan | RangeSelectExec: range_expr=[MIN(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host@1], time_index=ts | +| physical_plan | RangeSelectExec: range_expr=[min(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host@1], time_index=ts | |_|_CoalescePartitionsExec_| |_|_MergeScanExec: REDACTED |_|_| @@ -73,7 +73,7 @@ EXPLAIN ANALYZE SELECT ts, host, min(val) RANGE '5s' FROM host ALIGN '5s'; +-+-+-+ | stage | node | plan_| +-+-+-+ -| 0_| 0_|_RangeSelectExec: range_expr=[MIN(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host@1], time_index=ts REDACTED +| 0_| 0_|_RangeSelectExec: range_expr=[min(host.val) RANGE 5s], align=5000ms, align_to=0ms, align_by=[host@1], time_index=ts REDACTED |_|_|_CoalescePartitionsExec REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| diff --git a/tests/cases/standalone/common/range/precisions.result b/tests/cases/standalone/common/range/precisions.result index 1b54091c17..e7fd6ad22e 100644 --- a/tests/cases/standalone/common/range/precisions.result +++ b/tests/cases/standalone/common/range/precisions.result @@ -24,7 +24,7 @@ Affected Rows: 10 SELECT ts, host, min(val) RANGE '5s' FROM host_sec ALIGN '5s' ORDER BY host, ts; +---------------------+-------+----------------------------+ -| ts | host | MIN(host_sec.val) RANGE 5s | +| ts | host | min(host_sec.val) RANGE 5s | +---------------------+-------+----------------------------+ | 1970-01-01T00:00:00 | host1 | 0.0 | | 1970-01-01T00:00:05 | host1 | | diff --git a/tests/cases/standalone/common/range/special_aggr.result b/tests/cases/standalone/common/range/special_aggr.result index 1344d3bbdb..fa2b5cb47e 100644 --- a/tests/cases/standalone/common/range/special_aggr.result +++ b/tests/cases/standalone/common/range/special_aggr.result @@ -146,7 +146,7 @@ SELECT ts, host, first_value(addon ORDER BY val ASC, ts ASC) RANGE '5s', last_va SELECT ts, host, count(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+--------------------------+ -| ts | host | COUNT(host.val) RANGE 5s | +| ts | host | count(host.val) RANGE 5s | +---------------------+-------+--------------------------+ | 1970-01-01T00:00:00 | host1 | 3 | | 1970-01-01T00:00:05 | host1 | 0 | @@ -163,7 +163,7 @@ SELECT ts, host, count(val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; SELECT ts, host, count(distinct val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-----------------------------------+ -| ts | host | COUNT(DISTINCT host.val) RANGE 5s | +| ts | host | count(DISTINCT host.val) RANGE 5s | +---------------------+-------+-----------------------------------+ | 1970-01-01T00:00:00 | host1 | 3 | | 1970-01-01T00:00:05 | host1 | 0 | @@ -180,7 +180,7 @@ SELECT ts, host, count(distinct val) RANGE '5s' FROM host ALIGN '5s' ORDER BY ho SELECT ts, host, count(*) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+-------------------+ -| ts | host | COUNT(*) RANGE 5s | +| ts | host | count(*) RANGE 5s | +---------------------+-------+-------------------+ | 1970-01-01T00:00:00 | host1 | 3 | | 1970-01-01T00:00:05 | host1 | 3 | @@ -214,7 +214,7 @@ SELECT ts, host, count(1) RANGE '5s' as abc FROM host ALIGN '5s' ORDER BY host, SELECT ts, host, count(distinct *) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; +---------------------+-------+----------------------------+ -| ts | host | COUNT(DISTINCT *) RANGE 5s | +| ts | host | count(DISTINCT *) RANGE 5s | +---------------------+-------+----------------------------+ | 1970-01-01T00:00:00 | host1 | 3 | | 1970-01-01T00:00:05 | host1 | 3 | @@ -231,11 +231,9 @@ SELECT ts, host, count(distinct *) RANGE '5s' FROM host ALIGN '5s' ORDER BY host -- Test error first_value/last_value SELECT ts, host, first_value(val, val) RANGE '5s' FROM host ALIGN '5s' ORDER BY host, ts; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Error during planning: Coercion from [Int64, Int64] to the signature OneOf([ArraySignature(Array), Numeric(1), Uniform(1, [Utf8])]) failed. and No function matches the given name and argument types 'first_value(Int64, Int64)'. You might need to add explicit type casts. +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Error during planning: The function expected 1 arguments but received 2 No function matches the given name and argument types 'first_value(Int64, Int64)'. You might need to add explicit type casts. Candidate functions: - first_value(array) - first_value(Numeric(1)) - first_value(Utf8) + first_value(Any) DROP TABLE host; diff --git a/tests/cases/standalone/common/range/to.result b/tests/cases/standalone/common/range/to.result index e7e0445316..1488eca688 100644 --- a/tests/cases/standalone/common/range/to.result +++ b/tests/cases/standalone/common/range/to.result @@ -21,7 +21,7 @@ Affected Rows: 8 SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-23T00:00:00 | host1 | 0 | | 2024-01-24T00:00:00 | host1 | 2 | @@ -36,7 +36,7 @@ Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal `align SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '1900-01-01T00:00:00+01:00' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-22T23:00:00 | host1 | 0 | | 2024-01-23T23:00:00 | host1 | 1 | @@ -49,7 +49,7 @@ SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '1900-01-01T00:00:0 SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '2024-01-23T00:00:00+01:00' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-22T23:00:00 | host1 | 0 | | 2024-01-23T23:00:00 | host1 | 1 | @@ -62,7 +62,7 @@ SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '2024-01-23T00:00:0 SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '2023-01-01T00:00:00+01:00' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-22T23:00:00 | host1 | 0 | | 2024-01-23T23:00:00 | host1 | 1 | @@ -74,37 +74,37 @@ SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' TO '2023-01-01T00:00:0 SELECT ts, min(val) RANGE (INTERVAL '1' day) FROM host ALIGN (INTERVAL '1' day) TO '1900-01-01T00:00:00+01:00' by (1) ORDER BY ts; -+---------------------+------------------------------------------------------------------+ -| ts | MIN(host.val) RANGE IntervalMonthDayNano("18446744073709551616") | -+---------------------+------------------------------------------------------------------+ -| 2024-01-22T23:00:00 | 0 | -| 2024-01-23T23:00:00 | 1 | -| 2024-01-24T23:00:00 | 3 | -+---------------------+------------------------------------------------------------------+ ++---------------------+---------------------------------------------------------------------------------------------------------+ +| ts | min(host.val) RANGE IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") | ++---------------------+---------------------------------------------------------------------------------------------------------+ +| 2024-01-22T23:00:00 | 0 | +| 2024-01-23T23:00:00 | 1 | +| 2024-01-24T23:00:00 | 3 | ++---------------------+---------------------------------------------------------------------------------------------------------+ SELECT ts, min(val) RANGE (INTERVAL '2' day - INTERVAL '1' day) FROM host ALIGN (INTERVAL '2' day - INTERVAL '1' day) TO (now() - (now() + INTERVAL '1' hour)) by (1) ORDER BY ts; -+---------------------+-----------------------------------------------------------------------------------------------------------------+ -| ts | MIN(host.val) RANGE IntervalMonthDayNano("36893488147419103232") - IntervalMonthDayNano("18446744073709551616") | -+---------------------+-----------------------------------------------------------------------------------------------------------------+ -| 2024-01-22T23:00:00 | 0 | -| 2024-01-23T23:00:00 | 1 | -| 2024-01-24T23:00:00 | 3 | -+---------------------+-----------------------------------------------------------------------------------------------------------------+ ++---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| ts | min(host.val) RANGE IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 2, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") | ++---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2024-01-22T23:00:00 | 0 | +| 2024-01-23T23:00:00 | 1 | +| 2024-01-24T23:00:00 | 3 | ++---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -- non-positive duration SELECT ts, min(val) RANGE (INTERVAL '1' day - INTERVAL '2' day) FROM host ALIGN (INTERVAL '1' day) TO '1900-01-01T00:00:00+01:00' by (1) ORDER BY ts; -Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("18446744073709551616") - IntervalMonthDayNano("36893488147419103232")` in range select query +Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 2, nanoseconds: 0 }")` in range select query SELECT ts, min(val) RANGE (INTERVAL '1' day - INTERVAL '1' day) FROM host ALIGN (INTERVAL '1' day) TO '1900-01-01T00:00:00+01:00' by (1) ORDER BY ts; -Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("18446744073709551616") - IntervalMonthDayNano("18446744073709551616")` in range select query +Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }")` in range select query -- duration not all interval SELECT ts, min(val) RANGE (now() - INTERVAL '1' day) FROM host ALIGN (INTERVAL '1' day) TO '1900-01-01T00:00:00+01:00' by (1) ORDER BY ts; -Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `now() - IntervalMonthDayNano("18446744073709551616")` in range select query +Error: 3000(PlanQuery), DataFusion error: Error during planning: Illegal argument `now() - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }")` in range select query --- ALIGN TO with time zone --- set time_zone='Asia/Shanghai'; @@ -115,7 +115,7 @@ Affected Rows: 0 SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-23T08:00:00 | host1 | 0 | | 2024-01-24T08:00:00 | host1 | 2 | @@ -131,7 +131,7 @@ Affected Rows: 0 SELECT ts, host, min(val) RANGE '1d' FROM host ALIGN '1d' ORDER BY host, ts; +---------------------+-------+------------------------+ -| ts | host | MIN(host.val) RANGE 1d | +| ts | host | min(host.val) RANGE 1d | +---------------------+-------+------------------------+ | 2024-01-22T23:00:00 | host1 | 0 | | 2024-01-23T23:00:00 | host1 | 1 | diff --git a/tests/cases/standalone/common/select/flush_append_only.result b/tests/cases/standalone/common/select/flush_append_only.result index 38f221aa93..2413e8e0d4 100644 --- a/tests/cases/standalone/common/select/flush_append_only.result +++ b/tests/cases/standalone/common/select/flush_append_only.result @@ -67,7 +67,7 @@ from t; +-------------+ -| COUNT(t.ts) | +| count(t.ts) | +-------------+ | 18 | +-------------+ diff --git a/tests/cases/standalone/common/select/prune_field.result b/tests/cases/standalone/common/select/prune_field.result index d309d384be..6a19b67c21 100644 --- a/tests/cases/standalone/common/select/prune_field.result +++ b/tests/cases/standalone/common/select/prune_field.result @@ -1,13 +1,13 @@ CREATE TABLE IF NOT EXISTS prune_field ( ts TIMESTAMP TIME INDEX, - tag UInt16, + `tag` UInt16, a UInt8, b UInt8, -PRIMARY KEY (tag)) ENGINE = mito WITH('merge_mode'='last_non_null'); +PRIMARY KEY (`tag`)) ENGINE = mito WITH('merge_mode'='last_non_null'); Affected Rows: 0 -insert into prune_field(ts, tag, a, b) values(0, 1, 1, null); +insert into prune_field(ts, `tag`, a, b) values(0, 1, 1, null); Affected Rows: 1 @@ -19,7 +19,7 @@ admin flush_table('prune_field'); | 0 | +----------------------------------+ -insert into prune_field(ts, tag, a, b) values(0, 1, null, 1); +insert into prune_field(ts, `tag`, a, b) values(0, 1, null, 1); Affected Rows: 1 diff --git a/tests/cases/standalone/common/select/prune_field.sql b/tests/cases/standalone/common/select/prune_field.sql index 6488175508..a5cf255a0d 100644 --- a/tests/cases/standalone/common/select/prune_field.sql +++ b/tests/cases/standalone/common/select/prune_field.sql @@ -1,15 +1,15 @@ CREATE TABLE IF NOT EXISTS prune_field ( ts TIMESTAMP TIME INDEX, - tag UInt16, + `tag` UInt16, a UInt8, b UInt8, -PRIMARY KEY (tag)) ENGINE = mito WITH('merge_mode'='last_non_null'); +PRIMARY KEY (`tag`)) ENGINE = mito WITH('merge_mode'='last_non_null'); -insert into prune_field(ts, tag, a, b) values(0, 1, 1, null); +insert into prune_field(ts, `tag`, a, b) values(0, 1, 1, null); admin flush_table('prune_field'); -insert into prune_field(ts, tag, a, b) values(0, 1, null, 1); +insert into prune_field(ts, `tag`, a, b) values(0, 1, null, 1); admin flush_table('prune_field'); diff --git a/tests/cases/standalone/common/select/skipping_index.result b/tests/cases/standalone/common/select/skipping_index.result index a083073c33..20b3c40934 100644 --- a/tests/cases/standalone/common/select/skipping_index.result +++ b/tests/cases/standalone/common/select/skipping_index.result @@ -1,7 +1,7 @@ create table skipping_table ( ts timestamp time index, - id string skipping index, + `id` string skipping index, `name` string skipping index with (granularity = 8192), @@ -59,7 +59,6 @@ EXPLAIN ANALYZE SELECT * FROM skipping_table WHERE id = 'id2' ORDER BY `name`; | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| | 1_| 0_|_SortPreservingMergeExec: [name@2 ASC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_SortExec: expr=[name@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: id@1 = id2 REDACTED @@ -83,7 +82,6 @@ EXPLAIN ANALYZE SELECT * FROM skipping_table WHERE id = 'id5' ORDER BY `name`; | 0_| 0_|_MergeScanExec: REDACTED |_|_|_| | 1_| 0_|_SortPreservingMergeExec: [name@2 ASC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_SortExec: expr=[name@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: id@1 = id5 REDACTED diff --git a/tests/cases/standalone/common/select/skipping_index.sql b/tests/cases/standalone/common/select/skipping_index.sql index 9f9a38e25d..b17d70f598 100644 --- a/tests/cases/standalone/common/select/skipping_index.sql +++ b/tests/cases/standalone/common/select/skipping_index.sql @@ -1,7 +1,7 @@ create table skipping_table ( ts timestamp time index, - id string skipping index, + `id` string skipping index, `name` string skipping index with (granularity = 8192), diff --git a/tests/cases/standalone/common/show/show_create.result b/tests/cases/standalone/common/show/show_create.result index df7acddb7f..7a52c8181b 100644 --- a/tests/cases/standalone/common/show/show_create.result +++ b/tests/cases/standalone/common/show/show_create.result @@ -1,16 +1,16 @@ CREATE TABLE system_metrics ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT COMMENT 'comment', ts TIMESTAMP NOT NULL DEFAULT current_timestamp(), TIME INDEX (ts), - PRIMARY KEY (id, host) + PRIMARY KEY (`id`, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( @@ -91,18 +91,18 @@ drop table table_without_partition; Affected Rows: 0 CREATE TABLE not_supported_table_storage_option ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT, ts TIMESTAMP NOT NULL DEFAULT current_timestamp(), TIME INDEX (ts), - PRIMARY KEY (id, host) + PRIMARY KEY (`id`, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( diff --git a/tests/cases/standalone/common/show/show_create.sql b/tests/cases/standalone/common/show/show_create.sql index d7c9a30bda..c7a50a2598 100644 --- a/tests/cases/standalone/common/show/show_create.sql +++ b/tests/cases/standalone/common/show/show_create.sql @@ -1,16 +1,16 @@ CREATE TABLE system_metrics ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT COMMENT 'comment', ts TIMESTAMP NOT NULL DEFAULT current_timestamp(), TIME INDEX (ts), - PRIMARY KEY (id, host) + PRIMARY KEY (`id`, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( @@ -33,18 +33,18 @@ show create table table_without_partition; drop table table_without_partition; CREATE TABLE not_supported_table_storage_option ( - id INT UNSIGNED, + `id` INT UNSIGNED, host STRING, cpu DOUBLE, disk FLOAT, ts TIMESTAMP NOT NULL DEFAULT current_timestamp(), TIME INDEX (ts), - PRIMARY KEY (id, host) + PRIMARY KEY (`id`, host) ) -PARTITION ON COLUMNS (id) ( - id < 5, - id >= 5 AND id < 9, - id >= 9 +PARTITION ON COLUMNS (`id`) ( + `id` < 5, + `id` >= 5 AND `id` < 9, + `id` >= 9 ) ENGINE=mito WITH( diff --git a/tests/cases/standalone/common/subquery/table.result b/tests/cases/standalone/common/subquery/table.result index 549a387549..bcd107a35a 100644 --- a/tests/cases/standalone/common/subquery/table.result +++ b/tests/cases/standalone/common/subquery/table.result @@ -123,7 +123,7 @@ SELECT * FROM (SELECT i, j FROM test) AS a, (SELECT i+1 AS r,j FROM test) AS b, SELECT sum(x) FROM (SELECT i AS x FROM test GROUP BY i) sq; +-----------+ -| SUM(sq.x) | +| sum(sq.x) | +-----------+ | 12 | +-----------+ @@ -131,7 +131,7 @@ SELECT sum(x) FROM (SELECT i AS x FROM test GROUP BY i) sq; SELECT sum(x) FROM (SELECT i+1 AS x FROM test GROUP BY x) sq; +-----------+ -| SUM(sq.x) | +| sum(sq.x) | +-----------+ | 15 | +-----------+ diff --git a/tests/cases/standalone/common/subquery/test_neumann.result b/tests/cases/standalone/common/subquery/test_neumann.result index 6b42167953..4732156ce5 100644 --- a/tests/cases/standalone/common/subquery/test_neumann.result +++ b/tests/cases/standalone/common/subquery/test_neumann.result @@ -1,6 +1,6 @@ -- from: -- https://github.com/duckdb/duckdb/blob/74687ec572e9e6ccf34f9b15daa62998b34a3e13/test/sql/subquery/test_neumann.test -CREATE TABLE students(ts TIMESTAMP TIME INDEX, id INTEGER, n VARCHAR, major VARCHAR, y INTEGER); +CREATE TABLE students(ts TIMESTAMP TIME INDEX, `id` INTEGER, n VARCHAR, major VARCHAR, y INTEGER); Affected Rows: 0 @@ -32,7 +32,7 @@ INSERT INTO exams VALUES (4, 2, 'Graphics', 'CS', 7, 2016); Affected Rows: 1 -SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.id=e.sid AND e.grade=(SELECT MAX(e2.grade) FROM exams e2 WHERE s.id=e2.sid) ORDER BY n, course; +SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.`id`=e.sid AND e.grade=(SELECT MAX(e2.grade) FROM exams e2 WHERE s.`id`=e2.sid) ORDER BY n, course; +------+------------------+-------+ | n | course | grade | @@ -44,10 +44,10 @@ SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.id=e.sid AND e.gr -- skipped, unsupported feature: correlated column in predicate, see also: -- https://github.com/GreptimeTeam/greptimedb/issues/5012 --- SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.id=e.sid AND (s.major = 'CS' OR s.major = 'Games Eng') AND e.grade <= (SELECT AVG(e2.grade) - 1 FROM exams e2 WHERE s.id=e2.sid OR (e2.curriculum=s.major AND s.y>=e2.y)) ORDER BY n, course; +-- SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.`id`=e.sid AND (s.major = 'CS' OR s.major = 'Games Eng') AND e.grade <= (SELECT AVG(e2.grade) - 1 FROM exams e2 WHERE s.`id`=e2.sid OR (e2.curriculum=s.major AND s.y>=e2.y)) ORDER BY n, course; -- skipped, unsupported feature: exists, see also: -- https://github.com/GreptimeTeam/greptimedb/issues/5012 --- SELECT n, major FROM students s WHERE EXISTS(SELECT * FROM exams e WHERE e.sid=s.id AND grade=10) OR s.n='Dirk' ORDER BY n; +-- SELECT n, major FROM students s WHERE EXISTS(SELECT * FROM exams e WHERE e.sid=s.`id` AND grade=10) OR s.n='Dirk' ORDER BY n; DROP TABLE students; Affected Rows: 0 diff --git a/tests/cases/standalone/common/subquery/test_neumann.sql b/tests/cases/standalone/common/subquery/test_neumann.sql index 3b29b5be22..438172587f 100644 --- a/tests/cases/standalone/common/subquery/test_neumann.sql +++ b/tests/cases/standalone/common/subquery/test_neumann.sql @@ -1,6 +1,6 @@ -- from: -- https://github.com/duckdb/duckdb/blob/74687ec572e9e6ccf34f9b15daa62998b34a3e13/test/sql/subquery/test_neumann.test -CREATE TABLE students(ts TIMESTAMP TIME INDEX, id INTEGER, n VARCHAR, major VARCHAR, y INTEGER); +CREATE TABLE students(ts TIMESTAMP TIME INDEX, `id` INTEGER, n VARCHAR, major VARCHAR, y INTEGER); CREATE TABLE exams(ts TIMESTAMP TIME INDEX, sid INTEGER, course VARCHAR, curriculum VARCHAR, grade INTEGER, y INTEGER); @@ -16,15 +16,15 @@ INSERT INTO exams VALUES (3, 2, 'Database Systems', 'CS', 7, 2015); INSERT INTO exams VALUES (4, 2, 'Graphics', 'CS', 7, 2016); -SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.id=e.sid AND e.grade=(SELECT MAX(e2.grade) FROM exams e2 WHERE s.id=e2.sid) ORDER BY n, course; +SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.`id`=e.sid AND e.grade=(SELECT MAX(e2.grade) FROM exams e2 WHERE s.`id`=e2.sid) ORDER BY n, course; -- skipped, unsupported feature: correlated column in predicate, see also: -- https://github.com/GreptimeTeam/greptimedb/issues/5012 --- SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.id=e.sid AND (s.major = 'CS' OR s.major = 'Games Eng') AND e.grade <= (SELECT AVG(e2.grade) - 1 FROM exams e2 WHERE s.id=e2.sid OR (e2.curriculum=s.major AND s.y>=e2.y)) ORDER BY n, course; +-- SELECT s.n, e.course, e.grade FROM students s, exams e WHERE s.`id`=e.sid AND (s.major = 'CS' OR s.major = 'Games Eng') AND e.grade <= (SELECT AVG(e2.grade) - 1 FROM exams e2 WHERE s.`id`=e2.sid OR (e2.curriculum=s.major AND s.y>=e2.y)) ORDER BY n, course; -- skipped, unsupported feature: exists, see also: -- https://github.com/GreptimeTeam/greptimedb/issues/5012 --- SELECT n, major FROM students s WHERE EXISTS(SELECT * FROM exams e WHERE e.sid=s.id AND grade=10) OR s.n='Dirk' ORDER BY n; +-- SELECT n, major FROM students s WHERE EXISTS(SELECT * FROM exams e WHERE e.sid=s.`id` AND grade=10) OR s.n='Dirk' ORDER BY n; DROP TABLE students; diff --git a/tests/cases/standalone/common/system/information_schema.result b/tests/cases/standalone/common/system/information_schema.result index 10b5a9d0f7..18b13f0712 100644 --- a/tests/cases/standalone/common/system/information_schema.result +++ b/tests/cases/standalone/common/system/information_schema.result @@ -634,7 +634,7 @@ desc table build_info; select count(*) from build_info; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 1 | +----------+ diff --git a/tests/cases/standalone/common/tql-explain-analyze/analyze.result b/tests/cases/standalone/common/tql-explain-analyze/analyze.result index 5a08da5517..b40f30f2c1 100644 --- a/tests/cases/standalone/common/tql-explain-analyze/analyze.result +++ b/tests/cases/standalone/common/tql-explain-analyze/analyze.result @@ -26,9 +26,8 @@ TQL ANALYZE (0, 10, '5s') test; |_|_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST,j@1 DESC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_SortExec: expr=[k@2 DESC NULLS LAST,j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST, j@1 DESC NULLS LAST] REDACTED +|_|_|_SortExec: expr=[k@2 DESC NULLS LAST, j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: j@1 >= -300000 AND j@1 <= 310000 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED @@ -57,9 +56,8 @@ TQL ANALYZE (0, 10, '1s', '2s') test; |_|_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST,j@1 DESC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_SortExec: expr=[k@2 DESC NULLS LAST,j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST, j@1 DESC NULLS LAST] REDACTED +|_|_|_SortExec: expr=[k@2 DESC NULLS LAST, j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: j@1 >= -2000 AND j@1 <= 12000 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED @@ -87,9 +85,8 @@ TQL ANALYZE ('1970-01-01T00:00:00'::timestamp, '1970-01-01T00:00:00'::timestamp |_|_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST,j@1 DESC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_SortExec: expr=[k@2 DESC NULLS LAST,j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST, j@1 DESC NULLS LAST] REDACTED +|_|_|_SortExec: expr=[k@2 DESC NULLS LAST, j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: j@1 >= -300000 AND j@1 <= 310000 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED @@ -119,9 +116,8 @@ TQL ANALYZE VERBOSE (0, 10, '5s') test; |_|_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_MergeScanExec: REDACTED |_|_|_| -| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST,j@1 DESC NULLS LAST] REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED -|_|_|_SortExec: expr=[k@2 DESC NULLS LAST,j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED +| 1_| 0_|_SortPreservingMergeExec: [k@2 DESC NULLS LAST, j@1 DESC NULLS LAST] REDACTED +|_|_|_SortExec: expr=[k@2 DESC NULLS LAST, j@1 DESC NULLS LAST], preserve_partitioning=[true] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_FilterExec: j@1 >= -300000 AND j@1 <= 310000 REDACTED |_|_|_RepartitionExec: partitioning=REDACTED diff --git a/tests/cases/standalone/common/tql-explain-analyze/explain.result b/tests/cases/standalone/common/tql-explain-analyze/explain.result index ddfcc97ed1..68be4f7ee2 100644 --- a/tests/cases/standalone/common/tql-explain-analyze/explain.result +++ b/tests/cases/standalone/common/tql-explain-analyze/explain.result @@ -88,10 +88,11 @@ TQL EXPLAIN VERBOSE (0, 10, '5s') test; |_|_Sort: test.k DESC NULLS LAST, test.j DESC NULLS LAST_| |_|_Filter: test.j >= TimestampMillisecond(-300000, None) AND test.j <= TimestampMillisecond(300000, None)_| |_|_TableScan: test_| -| logical_plan after apply_function_rewrites_| SAME TEXT AS ABOVE_| | logical_plan after count_wildcard_to_time_index_rule_| SAME TEXT AS ABOVE_| | logical_plan after StringNormalizationRule_| SAME TEXT AS ABOVE_| | logical_plan after inline_table_scan_| SAME TEXT AS ABOVE_| +| logical_plan after expand_wildcard_rule_| SAME TEXT AS ABOVE_| +| logical_plan after resolve_grouping_function_| SAME TEXT AS ABOVE_| | logical_plan after type_coercion_| SAME TEXT AS ABOVE_| | logical_plan after DistPlannerAnalyzer_| PromInstantManipulate: range=[0..0], lookback=[300000], interval=[300000], time index=[j]_| |_|_PromSeriesNormalize: offset=[0], time index=[j], filter NaN: [false]_| @@ -106,8 +107,6 @@ TQL EXPLAIN VERBOSE (0, 10, '5s') test; | logical_plan after decorrelate_predicate_subquery_| SAME TEXT AS ABOVE_| | logical_plan after scalar_subquery_to_join_| SAME TEXT AS ABOVE_| | logical_plan after extract_equijoin_predicate_| SAME TEXT AS ABOVE_| -| logical_plan after simplify_expressions_| SAME TEXT AS ABOVE_| -| logical_plan after rewrite_disjunctive_predicate_| SAME TEXT AS ABOVE_| | logical_plan after eliminate_duplicated_expr_| SAME TEXT AS ABOVE_| | logical_plan after eliminate_filter_| SAME TEXT AS ABOVE_| | logical_plan after eliminate_cross_join_| SAME TEXT AS ABOVE_| @@ -140,6 +139,11 @@ TQL EXPLAIN VERBOSE (0, 10, '5s') test; |_|_PromSeriesDivideExec: tags=["k"], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]_| |_|_MergeScanExec: REDACTED |_|_| +| initial_physical_plan_with_schema_| PromInstantManipulateExec: range=[0..0], lookback=[300000], interval=[300000], time index=[j], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_PromSeriesNormalizeExec: offset=[0], time index=[j], filter NaN: [false], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_PromSeriesDivideExec: tags=["k"], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_MergeScanExec: REDACTED +|_|_| | physical_plan after parallelize_scan_| PromInstantManipulateExec: range=[0..0], lookback=[300000], interval=[300000], time index=[j]_| |_|_PromSeriesNormalizeExec: offset=[0], time index=[j], filter NaN: [false]_| |_|_PromSeriesDivideExec: tags=["k"]_| @@ -182,9 +186,10 @@ TQL EXPLAIN VERBOSE (0, 10, '5s') test; |_|_| | physical_plan after LimitAggregation_| SAME TEXT AS ABOVE_| | physical_plan after ProjectionPushdown_| SAME TEXT AS ABOVE_| +| physical_plan after LimitPushdown_| SAME TEXT AS ABOVE_| | physical_plan after WindowedSortRule_| SAME TEXT AS ABOVE_| | physical_plan after RemoveDuplicateRule_| SAME TEXT AS ABOVE_| -| physical_plan after PipelineChecker_| SAME TEXT AS ABOVE_| +| physical_plan after SanityCheckPlan_| SAME TEXT AS ABOVE_| | physical_plan_| PromInstantManipulateExec: range=[0..0], lookback=[300000], interval=[300000], time index=[j]_| |_|_PromSeriesNormalizeExec: offset=[0], time index=[j], filter NaN: [false]_| |_|_PromSeriesDivideExec: tags=["k"]_| @@ -199,6 +204,13 @@ TQL EXPLAIN VERBOSE (0, 10, '5s') test; |_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true], statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]_| |_|_MergeScanExec: REDACTED |_|_| +| physical_plan_with_schema_| PromInstantManipulateExec: range=[0..0], lookback=[300000], interval=[300000], time index=[j], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_PromSeriesNormalizeExec: offset=[0], time index=[j], filter NaN: [false], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_PromSeriesDivideExec: tags=["k"], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_SortPreservingMergeExec: [k@2 ASC NULLS LAST], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_SortExec: expr=[k@2 ASC NULLS LAST], preserve_partitioning=[true], schema=[i:Float64;N, j:Timestamp(Millisecond, None), k:Utf8;N]_| +|_|_MergeScanExec: REDACTED +|_|_| +-+-+ DROP TABLE test; diff --git a/tests/cases/standalone/common/tql/binary_operator.result b/tests/cases/standalone/common/tql/binary_operator.result index a3277d927b..47d524c952 100644 --- a/tests/cases/standalone/common/tql/binary_operator.result +++ b/tests/cases/standalone/common/tql/binary_operator.result @@ -116,7 +116,7 @@ Affected Rows: 12 tql eval (0, 30, '10s') (sum by(host) (irate(host_cpu_seconds_total{mode!="idle"}[1m0s])) / sum by (host)((irate(host_cpu_seconds_total[1m0s])))) * 100; +-------+---------------------+--------------------------------------------------------------------------------------+ -| host | ts | lhs.SUM(prom_irate(ts_range,val)) / rhs.SUM(prom_irate(ts_range,val)) * Float64(100) | +| host | ts | lhs.sum(prom_irate(ts_range,val)) / rhs.sum(prom_irate(ts_range,val)) * Float64(100) | +-------+---------------------+--------------------------------------------------------------------------------------+ | host1 | 1970-01-01T00:00:10 | 66.66666666666666 | | host1 | 1970-01-01T00:00:20 | 77.77777777777779 | diff --git a/tests/cases/standalone/common/tql/case_sensitive.result b/tests/cases/standalone/common/tql/case_sensitive.result index e4e305a27e..608140fafe 100644 --- a/tests/cases/standalone/common/tql/case_sensitive.result +++ b/tests/cases/standalone/common/tql/case_sensitive.result @@ -39,7 +39,7 @@ select table_name from information_schema.tables where table_type = 'BASE TABLE' tql eval (0,10,'5s') sum(MemAvailable / 4) + sum(MemTotal / 4); +---------------------+---------------------------------------------------------------------+ -| ts | MemAvailable.SUM(val / Float64(4)) + MemTotal.SUM(val / Float64(4)) | +| ts | MemAvailable.sum(val / Float64(4)) + MemTotal.sum(val / Float64(4)) | +---------------------+---------------------------------------------------------------------+ | 1970-01-01T00:00:00 | 62.5 | | 1970-01-01T00:00:05 | 67.5 | diff --git a/tests/cases/standalone/common/tql/join.result b/tests/cases/standalone/common/tql/join.result index e1b0b47dad..9fd2e31327 100644 --- a/tests/cases/standalone/common/tql/join.result +++ b/tests/cases/standalone/common/tql/join.result @@ -32,7 +32,7 @@ Affected Rows: 3 tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) + sum(prompt / 1000 * 0.0015); +---------------------+-----------------------------------------------------------------------------------------------------------+ -| ts | completion.SUM(val * Float64(0.0015) / Float64(1000)) + prompt.SUM(val / Float64(1000) * Float64(0.0015)) | +| ts | completion.sum(val * Float64(0.0015) / Float64(1000)) + prompt.sum(val / Float64(1000) * Float64(0.0015)) | +---------------------+-----------------------------------------------------------------------------------------------------------+ | 1970-01-01T00:00:00 | 0.000165 | | 1970-01-01T00:00:05 | 0.000495 | @@ -43,7 +43,7 @@ tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) + sum(prompt / 1000 * 0.00 tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) + sum(prompt * 0.0015 / 1000); +---------------------+-----------------------------------------------------------------------------------------------------------+ -| ts | completion.SUM(val * Float64(0.0015) / Float64(1000)) + prompt.SUM(val * Float64(0.0015) / Float64(1000)) | +| ts | completion.sum(val * Float64(0.0015) / Float64(1000)) + prompt.sum(val * Float64(0.0015) / Float64(1000)) | +---------------------+-----------------------------------------------------------------------------------------------------------+ | 1970-01-01T00:00:00 | 0.000165 | | 1970-01-01T00:00:05 | 0.000495 | @@ -54,7 +54,7 @@ tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) + sum(prompt * 0.0015 / 10 tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) by (model) + sum(prompt * 0.0015 / 1000) by (model); +---------+---------------------+-----------------------------------------------------------------------------------------------------------+ -| model | ts | completion.SUM(val * Float64(0.0015) / Float64(1000)) + prompt.SUM(val * Float64(0.0015) / Float64(1000)) | +| model | ts | completion.sum(val * Float64(0.0015) / Float64(1000)) + prompt.sum(val * Float64(0.0015) / Float64(1000)) | +---------+---------------------+-----------------------------------------------------------------------------------------------------------+ | model-a | 1970-01-01T00:00:00 | 0.000165 | | model-a | 1970-01-01T00:00:05 | 0.000165 | @@ -67,7 +67,7 @@ tql eval(0, 10, '5s') sum(completion * 0.0015 / 1000) by (model) + sum(prompt * tql eval(0, 10, '5s') sum(completion / 1000) + max(completion / 1000); +---------------------+-------------------------------------------------------------+ -| ts | lhs.SUM(val / Float64(1000)) + rhs.MAX(val / Float64(1000)) | +| ts | lhs.sum(val / Float64(1000)) + rhs.max(val / Float64(1000)) | +---------------------+-------------------------------------------------------------+ | 1970-01-01T00:00:00 | 0.02 | | 1970-01-01T00:00:05 | 0.05 | @@ -78,7 +78,7 @@ tql eval(0, 10, '5s') sum(completion / 1000) + max(completion / 1000); tql eval(0, 10, '5s') sum(completion / 1000) + sum(completion / 1000); +---------------------+-------------------------------------------------------------+ -| ts | lhs.SUM(val / Float64(1000)) + rhs.SUM(val / Float64(1000)) | +| ts | lhs.sum(val / Float64(1000)) + rhs.sum(val / Float64(1000)) | +---------------------+-------------------------------------------------------------+ | 1970-01-01T00:00:00 | 0.02 | | 1970-01-01T00:00:05 | 0.06 | diff --git a/tests/cases/standalone/common/types/decimal/decimal_aggregates.result b/tests/cases/standalone/common/types/decimal/decimal_aggregates.result index 3eab6d9541..b0f9932e76 100644 --- a/tests/cases/standalone/common/types/decimal/decimal_aggregates.result +++ b/tests/cases/standalone/common/types/decimal/decimal_aggregates.result @@ -29,7 +29,7 @@ SELECT MIN(NULL::DECIMAL), MIN('4567645908450368043562342564564938245.1'::DECIMAL(38,1))::VARCHAR; +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ -| MIN(NULL) | MIN(Utf8("0.1")) | MIN(Utf8("4938245.1")) | MIN(Utf8("45672564564938245.1")) | MIN(Utf8("4567645908450368043562342564564938245.1")) | +| min(NULL) | min(Utf8("0.1")) | min(Utf8("4938245.1")) | min(Utf8("45672564564938245.1")) | min(Utf8("4567645908450368043562342564564938245.1")) | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ | | 0.1 | 4938245.1 | 45672564564938245.1 | 4567645908450368043562342564564938245.1 | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ @@ -42,7 +42,7 @@ SELECT MAX(NULL::DECIMAL), MAX('4567645908450368043562342564564938245.1'::DECIMAL(38,1))::VARCHAR; +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ -| MAX(NULL) | MAX(Utf8("0.1")) | MAX(Utf8("4938245.1")) | MAX(Utf8("45672564564938245.1")) | MAX(Utf8("4567645908450368043562342564564938245.1")) | +| max(NULL) | max(Utf8("0.1")) | max(Utf8("4938245.1")) | max(Utf8("45672564564938245.1")) | max(Utf8("4567645908450368043562342564564938245.1")) | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ | | 0.1 | 4938245.1 | 45672564564938245.1 | 4567645908450368043562342564564938245.1 | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ @@ -55,7 +55,7 @@ SELECT SUM(NULL::DECIMAL), SUM('4567645908450368043562342564564938245.1'::DECIMAL(38,1))::VARCHAR; +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ -| SUM(NULL) | SUM(Utf8("0.1")) | SUM(Utf8("4938245.1")) | SUM(Utf8("45672564564938245.1")) | SUM(Utf8("4567645908450368043562342564564938245.1")) | +| sum(NULL) | sum(Utf8("0.1")) | sum(Utf8("4938245.1")) | sum(Utf8("45672564564938245.1")) | sum(Utf8("4567645908450368043562342564564938245.1")) | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ | | 0.1 | 4938245.1 | 45672564564938245.1 | 4567645908450368043562342564564938245.1 | +-----------+------------------+------------------------+----------------------------------+------------------------------------------------------+ @@ -81,7 +81,7 @@ Affected Rows: 3 SELECT SUM(d1)::VARCHAR, SUM(d2)::VARCHAR, SUM(d3)::VARCHAR, SUM(d4)::VARCHAR FROM decimals; +------------------+------------------+------------------+------------------+ -| SUM(decimals.d1) | SUM(decimals.d2) | SUM(decimals.d3) | SUM(decimals.d4) | +| sum(decimals.d1) | sum(decimals.d2) | sum(decimals.d3) | sum(decimals.d4) | +------------------+------------------+------------------+------------------+ | 1368.0 | 845586.0 | 587848752.0 | 430998662178.0 | +------------------+------------------+------------------+------------------+ @@ -93,7 +93,7 @@ Affected Rows: 2 SELECT SUM(d1)::VARCHAR, SUM(d2)::VARCHAR, SUM(d3)::VARCHAR, SUM(d4)::VARCHAR FROM decimals; +------------------+------------------+------------------+------------------+ -| SUM(decimals.d1) | SUM(decimals.d2) | SUM(decimals.d3) | SUM(decimals.d4) | +| sum(decimals.d1) | sum(decimals.d2) | sum(decimals.d3) | sum(decimals.d4) | +------------------+------------------+------------------+------------------+ | 1368.3 | 845586.3 | 587848752.3 | 430998662178.3 | +------------------+------------------+------------------+------------------+ diff --git a/tests/cases/standalone/common/types/decimal/decimal_arithmetic.result b/tests/cases/standalone/common/types/decimal/decimal_arithmetic.result index 28a5388ca3..606a1b1f4d 100644 --- a/tests/cases/standalone/common/types/decimal/decimal_arithmetic.result +++ b/tests/cases/standalone/common/types/decimal/decimal_arithmetic.result @@ -67,10 +67,8 @@ SELECT ('0.5'::DECIMAL(1,1) + 10000)::VARCHAR, +----------------------------+--------------------------------+-------------------------------------+------------------------------------------+-----------------------------------------------+----------------------------------------------------+ -- out of range -SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 10000)::VARCHAR; - -Error: 3001(EngineExecuteQuery), Compute error: Overflow happened on: 10000 * 100000000000000000000000000000000000 - +-- Wait for https://github.com/apache/datafusion/pull/14126 +-- SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 10000)::VARCHAR; -- different types SELECT '0.5'::DECIMAL(1,1) + 1::TINYINT, '0.5'::DECIMAL(1,1) + 2::SMALLINT, diff --git a/tests/cases/standalone/common/types/decimal/decimal_arithmetic.sql b/tests/cases/standalone/common/types/decimal/decimal_arithmetic.sql index f8afe63efe..cd8dba86da 100644 --- a/tests/cases/standalone/common/types/decimal/decimal_arithmetic.sql +++ b/tests/cases/standalone/common/types/decimal/decimal_arithmetic.sql @@ -33,7 +33,8 @@ SELECT ('0.5'::DECIMAL(1,1) + 10000)::VARCHAR, -- out of range -SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 10000)::VARCHAR; +-- Wait for https://github.com/apache/datafusion/pull/14126 +-- SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 10000)::VARCHAR; -- different types diff --git a/tests/cases/standalone/common/types/decimal/decimal_cast.result b/tests/cases/standalone/common/types/decimal/decimal_cast.result index 5f9bb7955e..ec18623233 100644 --- a/tests/cases/standalone/common/types/decimal/decimal_cast.result +++ b/tests/cases/standalone/common/types/decimal/decimal_cast.result @@ -124,7 +124,7 @@ Error: 3001(EngineExecuteQuery), Invalid argument error: 1000000000 is too large SELECT 100::TINYINT::DECIMAL(38,37); -Error: 3001(EngineExecuteQuery), Compute error: Overflow happened on: 100 * 10000000000000000000000000000000000000 +Error: 3001(EngineExecuteQuery), Arithmetic overflow: Overflow happened on: 100 * 10000000000000000000000000000000000000 -- smallint SELECT 100::SMALLINT::DECIMAL(18,3), 200::SMALLINT::DECIMAL(3,0), (-300)::SMALLINT::DECIMAL(3,0), 0::SMALLINT::DECIMAL(3,3); @@ -162,7 +162,7 @@ Error: 3001(EngineExecuteQuery), Invalid argument error: 1000000000 is too large SELECT 100::SMALLINT::DECIMAL(38,37); -Error: 3001(EngineExecuteQuery), Compute error: Overflow happened on: 100 * 10000000000000000000000000000000000000 +Error: 3001(EngineExecuteQuery), Arithmetic overflow: Overflow happened on: 100 * 10000000000000000000000000000000000000 -- integer SELECT 100::INTEGER::DECIMAL(18,3), 200::INTEGER::DECIMAL(3,0), (-300)::INTEGER::DECIMAL(3,0), 0::INTEGER::DECIMAL(3,3); @@ -208,7 +208,7 @@ Error: 3001(EngineExecuteQuery), Invalid argument error: 1000000000 is too large SELECT 100::INTEGER::DECIMAL(38,37); -Error: 3001(EngineExecuteQuery), Compute error: Overflow happened on: 100 * 10000000000000000000000000000000000000 +Error: 3001(EngineExecuteQuery), Arithmetic overflow: Overflow happened on: 100 * 10000000000000000000000000000000000000 -- bigint SELECT 100::BIGINT::DECIMAL(18,3), 200::BIGINT::DECIMAL(3,0), (-100)::BIGINT::DECIMAL(3,0), 0::BIGINT::DECIMAL(3,3); @@ -262,7 +262,7 @@ Error: 3001(EngineExecuteQuery), Invalid argument error: 1000000000 is too large SELECT 100::BIGINT::DECIMAL(38,37); -Error: 3001(EngineExecuteQuery), Compute error: Overflow happened on: 100 * 10000000000000000000000000000000000000 +Error: 3001(EngineExecuteQuery), Arithmetic overflow: Overflow happened on: 100 * 10000000000000000000000000000000000000 -- float SELECT 100::FLOAT::DECIMAL(18,3), 200::FLOAT::DECIMAL(3,0), (-300)::FLOAT::DECIMAL(3,0), 0::FLOAT::DECIMAL(3,3); diff --git a/tests/cases/standalone/common/types/float/nan_aggregate.result b/tests/cases/standalone/common/types/float/nan_aggregate.result index 77636174c5..2d191ead10 100644 --- a/tests/cases/standalone/common/types/float/nan_aggregate.result +++ b/tests/cases/standalone/common/types/float/nan_aggregate.result @@ -11,7 +11,7 @@ Affected Rows: 6 SELECT f, SUM(i) FROM floats GROUP BY f ORDER BY f; +------+---------------+ -| f | SUM(floats.i) | +| f | sum(floats.i) | +------+---------------+ | -inf | 5 | | inf | 8 | @@ -21,7 +21,7 @@ SELECT f, SUM(i) FROM floats GROUP BY f ORDER BY f; SELECT SUM(f) FROM floats WHERE f > 0 AND f != 'nan'::FLOAT; +---------------+ -| SUM(floats.f) | +| sum(floats.f) | +---------------+ | inf | +---------------+ @@ -29,7 +29,7 @@ SELECT SUM(f) FROM floats WHERE f > 0 AND f != 'nan'::FLOAT; SELECT SUM(f) FROM floats WHERE f < 0; +---------------+ -| SUM(floats.f) | +| sum(floats.f) | +---------------+ | -inf | +---------------+ @@ -37,7 +37,7 @@ SELECT SUM(f) FROM floats WHERE f < 0; SELECT SUM(f) FROM floats; +---------------+ -| SUM(floats.f) | +| sum(floats.f) | +---------------+ | NaN | +---------------+ @@ -58,7 +58,7 @@ Affected Rows: 6 SELECT f, SUM(i) FROM doubles GROUP BY f ORDER BY f; +------+----------------+ -| f | SUM(doubles.i) | +| f | sum(doubles.i) | +------+----------------+ | -inf | 5 | | inf | 8 | @@ -68,7 +68,7 @@ SELECT f, SUM(i) FROM doubles GROUP BY f ORDER BY f; SELECT SUM(f) FROM doubles WHERE f > 0 AND f != 'nan'::DOUBLE; +----------------+ -| SUM(doubles.f) | +| sum(doubles.f) | +----------------+ | inf | +----------------+ @@ -76,7 +76,7 @@ SELECT SUM(f) FROM doubles WHERE f > 0 AND f != 'nan'::DOUBLE; SELECT SUM(f) FROM doubles WHERE f < 0; +----------------+ -| SUM(doubles.f) | +| sum(doubles.f) | +----------------+ | -inf | +----------------+ @@ -84,7 +84,7 @@ SELECT SUM(f) FROM doubles WHERE f < 0; SELECT SUM(f) FROM doubles; +----------------+ -| SUM(doubles.f) | +| sum(doubles.f) | +----------------+ | NaN | +----------------+ @@ -117,7 +117,7 @@ SELECT * FROM floats_doubles; SELECT SUM(f) FROM floats_doubles WHERE f > 0; +-----------------------+ -| SUM(floats_doubles.f) | +| sum(floats_doubles.f) | +-----------------------+ | 3.999999872114277e38 | +-----------------------+ @@ -126,7 +126,7 @@ SELECT SUM(f) FROM floats_doubles WHERE f > 0; SELECT SUM(d) FROM floats_doubles WHERE d > 0; +-----------------------+ -| SUM(floats_doubles.d) | +| sum(floats_doubles.d) | +-----------------------+ | inf | +-----------------------+ diff --git a/tests/cases/standalone/common/types/float/nan_aggregates.result b/tests/cases/standalone/common/types/float/nan_aggregates.result index 5dee887f23..60ba2da99b 100644 --- a/tests/cases/standalone/common/types/float/nan_aggregates.result +++ b/tests/cases/standalone/common/types/float/nan_aggregates.result @@ -3,7 +3,7 @@ SELECT SUM('inf'::FLOAT), SUM('-inf'::FLOAT), SUM('nan'::FLOAT); +------------------+-------------------+------------------+ -| SUM(Utf8("inf")) | SUM(Utf8("-inf")) | SUM(Utf8("nan")) | +| sum(Utf8("inf")) | sum(Utf8("-inf")) | sum(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -11,7 +11,7 @@ SELECT SUM('inf'::FLOAT), SUM('-inf'::FLOAT), SUM('nan'::FLOAT); SELECT MIN('inf'::FLOAT), MIN('-inf'::FLOAT), MIN('nan'::FLOAT); +------------------+-------------------+------------------+ -| MIN(Utf8("inf")) | MIN(Utf8("-inf")) | MIN(Utf8("nan")) | +| min(Utf8("inf")) | min(Utf8("-inf")) | min(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -19,7 +19,7 @@ SELECT MIN('inf'::FLOAT), MIN('-inf'::FLOAT), MIN('nan'::FLOAT); SELECT MAX('inf'::FLOAT), MAX('-inf'::FLOAT), MAX('nan'::FLOAT); +------------------+-------------------+------------------+ -| MAX(Utf8("inf")) | MAX(Utf8("-inf")) | MAX(Utf8("nan")) | +| max(Utf8("inf")) | max(Utf8("-inf")) | max(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -35,7 +35,7 @@ Affected Rows: 3 SELECT MIN(f), MAX(f) FROM floats; +---------------+---------------+ -| MIN(floats.f) | MAX(floats.f) | +| min(floats.f) | max(floats.f) | +---------------+---------------+ | -inf | NaN | +---------------+---------------+ @@ -48,7 +48,7 @@ Affected Rows: 0 SELECT SUM('inf'::DOUBLE), SUM('-inf'::DOUBLE), SUM('nan'::DOUBLE); +------------------+-------------------+------------------+ -| SUM(Utf8("inf")) | SUM(Utf8("-inf")) | SUM(Utf8("nan")) | +| sum(Utf8("inf")) | sum(Utf8("-inf")) | sum(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -56,7 +56,7 @@ SELECT SUM('inf'::DOUBLE), SUM('-inf'::DOUBLE), SUM('nan'::DOUBLE); SELECT MIN('inf'::DOUBLE), MIN('-inf'::DOUBLE), MIN('nan'::DOUBLE); +------------------+-------------------+------------------+ -| MIN(Utf8("inf")) | MIN(Utf8("-inf")) | MIN(Utf8("nan")) | +| min(Utf8("inf")) | min(Utf8("-inf")) | min(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -64,7 +64,7 @@ SELECT MIN('inf'::DOUBLE), MIN('-inf'::DOUBLE), MIN('nan'::DOUBLE); SELECT MAX('inf'::DOUBLE), MAX('-inf'::DOUBLE), MAX('nan'::DOUBLE); +------------------+-------------------+------------------+ -| MAX(Utf8("inf")) | MAX(Utf8("-inf")) | MAX(Utf8("nan")) | +| max(Utf8("inf")) | max(Utf8("-inf")) | max(Utf8("nan")) | +------------------+-------------------+------------------+ | inf | -inf | NaN | +------------------+-------------------+------------------+ @@ -80,7 +80,7 @@ Affected Rows: 3 SELECT MIN(f), MAX(f) FROM doubles; +----------------+----------------+ -| MIN(doubles.f) | MAX(doubles.f) | +| min(doubles.f) | max(doubles.f) | +----------------+----------------+ | -inf | NaN | +----------------+----------------+ diff --git a/tests/cases/standalone/common/types/float/nan_ordering.result b/tests/cases/standalone/common/types/float/nan_ordering.result index 4d3b495a3d..0ecfe612a4 100644 --- a/tests/cases/standalone/common/types/float/nan_ordering.result +++ b/tests/cases/standalone/common/types/float/nan_ordering.result @@ -81,7 +81,7 @@ SELECT f FROM floats ORDER BY f NULLS LAST LIMIT 4; SELECT COUNT(*) FROM floats WHERE f > 0; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 3 | +----------+ @@ -89,7 +89,7 @@ SELECT COUNT(*) FROM floats WHERE f > 0; SELECT COUNT(*) FROM floats WHERE f < 0; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 2 | +----------+ @@ -180,7 +180,7 @@ SELECT d FROM doubles ORDER BY d NULLS LAST LIMIT 4; SELECT COUNT(*) FROM doubles WHERE d > 0; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 3 | +----------+ @@ -188,7 +188,7 @@ SELECT COUNT(*) FROM doubles WHERE d > 0; SELECT COUNT(*) FROM doubles WHERE d < 0; +----------+ -| COUNT(*) | +| count(*) | +----------+ | 2 | +----------+ diff --git a/tests/cases/standalone/common/types/float/nan_window.result b/tests/cases/standalone/common/types/float/nan_window.result index 9af92a2098..6c736bd9da 100644 --- a/tests/cases/standalone/common/types/float/nan_window.result +++ b/tests/cases/standalone/common/types/float/nan_window.result @@ -11,7 +11,7 @@ Affected Rows: 6 SELECT f, SUM(i) OVER (PARTITION BY f) FROM floats ORDER BY f; +------+------------------------------------------------------------------------------------------------+ -| f | SUM(floats.i) PARTITION BY [floats.f] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING | +| f | sum(floats.i) PARTITION BY [floats.f] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING | +------+------------------------------------------------------------------------------------------------+ | -inf | 5 | | -inf | 5 | @@ -24,7 +24,7 @@ SELECT f, SUM(i) OVER (PARTITION BY f) FROM floats ORDER BY f; SELECT f, i, SUM(i) OVER (ORDER BY f, i) FROM floats ORDER BY f, i; +------+----+-----------------------------------------------------------------------------------------------------------------------------+ -| f | i | SUM(floats.i) ORDER BY [floats.f ASC NULLS LAST, floats.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| f | i | sum(floats.i) ORDER BY [floats.f ASC NULLS LAST, floats.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +------+----+-----------------------------------------------------------------------------------------------------------------------------+ | -inf | 2 | 2 | | -inf | 3 | 5 | @@ -37,7 +37,7 @@ SELECT f, i, SUM(i) OVER (ORDER BY f, i) FROM floats ORDER BY f, i; SELECT f, i, SUM(i) OVER (PARTITION BY f ORDER BY f, i) FROM floats ORDER BY f, i; +------+----+-----------------------------------------------------------------------------------------------------------------------------------------------------+ -| f | i | SUM(floats.i) PARTITION BY [floats.f] ORDER BY [floats.f ASC NULLS LAST, floats.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| f | i | sum(floats.i) PARTITION BY [floats.f] ORDER BY [floats.f ASC NULLS LAST, floats.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +------+----+-----------------------------------------------------------------------------------------------------------------------------------------------------+ | -inf | 2 | 2 | | -inf | 3 | 5 | @@ -50,7 +50,7 @@ SELECT f, i, SUM(i) OVER (PARTITION BY f ORDER BY f, i) FROM floats ORDER BY f, SELECT i, f, SUM(i) OVER (ORDER BY i, f) FROM floats ORDER BY i, f; +----+------+-----------------------------------------------------------------------------------------------------------------------------+ -| i | f | SUM(floats.i) ORDER BY [floats.i ASC NULLS LAST, floats.f ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| i | f | sum(floats.i) ORDER BY [floats.i ASC NULLS LAST, floats.f ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +----+------+-----------------------------------------------------------------------------------------------------------------------------+ | 1 | inf | 1 | | 2 | -inf | 3 | @@ -76,7 +76,7 @@ Affected Rows: 6 SELECT f, SUM(i) OVER (PARTITION BY f) FROM doubles ORDER BY f; +------+--------------------------------------------------------------------------------------------------+ -| f | SUM(doubles.i) PARTITION BY [doubles.f] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING | +| f | sum(doubles.i) PARTITION BY [doubles.f] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING | +------+--------------------------------------------------------------------------------------------------+ | -inf | 5 | | -inf | 5 | @@ -89,7 +89,7 @@ SELECT f, SUM(i) OVER (PARTITION BY f) FROM doubles ORDER BY f; SELECT f, i, SUM(i) OVER (ORDER BY f, i) FROM doubles ORDER BY f, i; +------+----+--------------------------------------------------------------------------------------------------------------------------------+ -| f | i | SUM(doubles.i) ORDER BY [doubles.f ASC NULLS LAST, doubles.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| f | i | sum(doubles.i) ORDER BY [doubles.f ASC NULLS LAST, doubles.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +------+----+--------------------------------------------------------------------------------------------------------------------------------+ | -inf | 2 | 2 | | -inf | 3 | 5 | @@ -102,7 +102,7 @@ SELECT f, i, SUM(i) OVER (ORDER BY f, i) FROM doubles ORDER BY f, i; SELECT f, i, SUM(i) OVER (PARTITION BY f ORDER BY f, i) FROM doubles ORDER BY f, i; +------+----+---------------------------------------------------------------------------------------------------------------------------------------------------------+ -| f | i | SUM(doubles.i) PARTITION BY [doubles.f] ORDER BY [doubles.f ASC NULLS LAST, doubles.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| f | i | sum(doubles.i) PARTITION BY [doubles.f] ORDER BY [doubles.f ASC NULLS LAST, doubles.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +------+----+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | -inf | 2 | 2 | | -inf | 3 | 5 | @@ -115,7 +115,7 @@ SELECT f, i, SUM(i) OVER (PARTITION BY f ORDER BY f, i) FROM doubles ORDER BY f, SELECT i, f, SUM(i) OVER (ORDER BY i, f) FROM doubles ORDER BY i, f; +----+------+--------------------------------------------------------------------------------------------------------------------------------+ -| i | f | SUM(doubles.i) ORDER BY [doubles.i ASC NULLS LAST, doubles.f ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| i | f | sum(doubles.i) ORDER BY [doubles.i ASC NULLS LAST, doubles.f ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +----+------+--------------------------------------------------------------------------------------------------------------------------------+ | 1 | inf | 1 | | 2 | -inf | 3 | diff --git a/tests/cases/standalone/common/types/interval/interval.result b/tests/cases/standalone/common/types/interval/interval.result index e35a426d37..56531ffd4d 100644 --- a/tests/cases/standalone/common/types/interval/interval.result +++ b/tests/cases/standalone/common/types/interval/interval.result @@ -1,71 +1,75 @@ -- common test SELECT INTERVAL '1 year 2 months 3 days 4 hours 5 minutes 6 seconds 100 microseconds'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("1109194275255040973236744059552") | -+---------------------------------------------------------+ -| 0 years 14 mons 3 days 4 hours 5 mins 6.000100000 secs | -+---------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 14706000100000 }") | ++---------------------------------------------------------------------------------------------------+ +| 14 mons 3 days 4 hours 5 mins 6.000100000 secs | ++---------------------------------------------------------------------------------------------------+ SELECT INTERVAL '1.5 year'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("1426106925256758076683791106048") | -+---------------------------------------------------------+ -| 0 years 18 mons 0 days 0 hours 0 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 18, days: 0, nanoseconds: 0 }") | ++--------------------------------------------------------------------------------------+ +| 18 mons | ++--------------------------------------------------------------------------------------+ SELECT INTERVAL '-2 months'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("-158456325028528675187087900672") | -+---------------------------------------------------------+ -| 0 years -2 mons 0 days 0 hours 0 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: -2, days: 0, nanoseconds: 0 }") | ++--------------------------------------------------------------------------------------+ +| -2 mons | ++--------------------------------------------------------------------------------------+ SELECT '3 hours'::INTERVAL; -+-------------------------------------------------------+ -| IntervalMonthDayNano("10800000000000") | -+-------------------------------------------------------+ -| 0 years 0 mons 0 days 3 hours 0 mins 0.000000000 secs | -+-------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 10800000000000 }") | ++--------------------------------------------------------------------------------------------------+ +| 3 hours | ++--------------------------------------------------------------------------------------------------+ SELECT INTERVAL '1 year 2 months 3 days 4 hours' + INTERVAL '1 year'; -+------------------------------------------------------------------------------------------------------------------+ -| IntervalMonthDayNano("1109194275255040972930743959552") + IntervalMonthDayNano("950737950171172051122527404032") | -+------------------------------------------------------------------------------------------------------------------+ -| 0 years 26 mons 3 days 4 hours 0 mins 0.000000000 secs | -+------------------------------------------------------------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 14400000000000 }") + IntervalMonthDayNano("IntervalMonthDayNano { months: 12, days: 0, nanoseconds: 0 }") | ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 26 mons 3 days 4 hours | ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT INTERVAL '1 year 2 months 3 days 4 hours' - INTERVAL '1 year'; -+------------------------------------------------------------------------------------------------------------------+ -| IntervalMonthDayNano("1109194275255040972930743959552") - IntervalMonthDayNano("950737950171172051122527404032") | -+------------------------------------------------------------------------------------------------------------------+ -| 0 years 2 mons 3 days 4 hours 0 mins 0.000000000 secs | -+------------------------------------------------------------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 14, days: 3, nanoseconds: 14400000000000 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 12, days: 0, nanoseconds: 0 }") | ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 2 mons 3 days 4 hours | ++------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT INTERVAL '6 years' * 2; -Error: 1001(Unsupported), Failed to plan SQL: This feature is not implemented: Unsupported interval operator: Multiply +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Cannot coerce arithmetic expression Interval(MonthDayNano) * Int64 to valid types SELECT INTERVAL '6 years' / 2; -Error: 1001(Unsupported), Failed to plan SQL: This feature is not implemented: Unsupported interval operator: Divide +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Cannot coerce arithmetic expression Interval(MonthDayNano) / Int64 to valid types SELECT INTERVAL '6 years' = INTERVAL '72 months'; -Error: 1001(Unsupported), Failed to plan SQL: This feature is not implemented: Unsupported interval operator: Eq ++-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 72, days: 0, nanoseconds: 0 }") = IntervalMonthDayNano("IntervalMonthDayNano { months: 72, days: 0, nanoseconds: 0 }") | ++-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| true | ++-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT arrow_typeof(INTERVAL '1 month'); -+---------------------------------------------------------------------+ -| arrow_typeof(IntervalMonthDayNano("79228162514264337593543950336")) | -+---------------------------------------------------------------------+ -| Interval(MonthDayNano) | -+---------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------+ +| arrow_typeof(IntervalMonthDayNano("IntervalMonthDayNano { months: 1, days: 0, nanoseconds: 0 }")) | ++---------------------------------------------------------------------------------------------------+ +| Interval(MonthDayNano) | ++---------------------------------------------------------------------------------------------------+ -- INTERVAL + TIME CONSTANT SELECT current_time() + INTERVAL '1 hour'; @@ -109,42 +113,42 @@ Affected Rows: 13 SELECT * FROM intervals; -+---------------------+-------------------------------------------------------------+ -| ts | interval_value | -+---------------------+-------------------------------------------------------------+ -| 2022-01-01T00:00:01 | 0 years 12 mons 0 days 0 hours 0 mins 0.000000000 secs | -| 2022-01-01T00:00:02 | 0 years 12 mons 0 days 0 hours 0 mins 0.000000000 secs | -| 2022-02-01T00:00:01 | 0 years 26 mons 0 days 0 hours 0 mins 0.000000000 secs | -| 2022-03-01T00:00:01 | 0 years 36 mons 0 days 3 hours 0 mins 0.000000000 secs | -| 2022-04-01T00:00:01 | 0 years 48 mons 0 days 0 hours 4 mins 0.000000000 secs | -| 2022-05-01T00:00:01 | 0 years 60 mons 0 days 0 hours 0 mins 5.000000000 secs | -| 2022-06-01T00:00:01 | 0 years 72 mons 0 days 0 hours 0 mins 0.006000000 secs | -| 2022-07-01T00:00:01 | 0 years 84 mons 0 days 0 hours 0 mins 0.000007000 secs | -| 2022-08-01T00:00:01 | 0 years 96 mons 0 days 0 hours 0 mins 0.000000008 secs | -| 2022-09-01T00:00:01 | 0 years 108 mons 9 days 0 hours 0 mins 0.000000000 secs | -| 2022-10-01T00:00:01 | 0 years 120 mons 0 days 10 hours 10 mins 10.010010010 secs | -| 2022-11-01T00:00:01 | 0 years 132 mons 11 days 11 hours 11 mins 11.011011011 secs | -| 2022-12-01T00:00:01 | 0 years 144 mons 12 days 12 hours 12 mins 12.012012012 secs | -+---------------------+-------------------------------------------------------------+ ++---------------------+-----------------------------------------------------+ +| ts | interval_value | ++---------------------+-----------------------------------------------------+ +| 2022-01-01T00:00:01 | 12 mons | +| 2022-01-01T00:00:02 | 12 mons | +| 2022-02-01T00:00:01 | 26 mons | +| 2022-03-01T00:00:01 | 36 mons 3 hours | +| 2022-04-01T00:00:01 | 48 mons 4 mins | +| 2022-05-01T00:00:01 | 60 mons 5.000000000 secs | +| 2022-06-01T00:00:01 | 72 mons 0.006000000 secs | +| 2022-07-01T00:00:01 | 84 mons 0.000007000 secs | +| 2022-08-01T00:00:01 | 96 mons 0.000000008 secs | +| 2022-09-01T00:00:01 | 108 mons 9 days | +| 2022-10-01T00:00:01 | 120 mons 10 hours 10 mins 10.010010010 secs | +| 2022-11-01T00:00:01 | 132 mons 11 days 11 hours 11 mins 11.011011011 secs | +| 2022-12-01T00:00:01 | 144 mons 12 days 12 hours 12 mins 12.012012012 secs | ++---------------------+-----------------------------------------------------+ SELECT DISTINCT interval_value FROM intervals ORDER BY interval_value; -+-------------------------------------------------------------+ -| interval_value | -+-------------------------------------------------------------+ -| 0 years 12 mons 0 days 0 hours 0 mins 0.000000000 secs | -| 0 years 26 mons 0 days 0 hours 0 mins 0.000000000 secs | -| 0 years 36 mons 0 days 3 hours 0 mins 0.000000000 secs | -| 0 years 48 mons 0 days 0 hours 4 mins 0.000000000 secs | -| 0 years 60 mons 0 days 0 hours 0 mins 5.000000000 secs | -| 0 years 72 mons 0 days 0 hours 0 mins 0.006000000 secs | -| 0 years 84 mons 0 days 0 hours 0 mins 0.000007000 secs | -| 0 years 96 mons 0 days 0 hours 0 mins 0.000000008 secs | -| 0 years 108 mons 9 days 0 hours 0 mins 0.000000000 secs | -| 0 years 120 mons 0 days 10 hours 10 mins 10.010010010 secs | -| 0 years 132 mons 11 days 11 hours 11 mins 11.011011011 secs | -| 0 years 144 mons 12 days 12 hours 12 mins 12.012012012 secs | -+-------------------------------------------------------------+ ++-----------------------------------------------------+ +| interval_value | ++-----------------------------------------------------+ +| 12 mons | +| 26 mons | +| 36 mons 3 hours | +| 48 mons 4 mins | +| 60 mons 5.000000000 secs | +| 72 mons 0.006000000 secs | +| 84 mons 0.000007000 secs | +| 96 mons 0.000000008 secs | +| 108 mons 9 days | +| 120 mons 10 hours 10 mins 10.010010010 secs | +| 132 mons 11 days 11 hours 11 mins 11.011011011 secs | +| 144 mons 12 days 12 hours 12 mins 12.012012012 secs | ++-----------------------------------------------------+ -- ts + interval SELECT ts + interval_value as new_value from intervals; @@ -275,123 +279,129 @@ SELECT TIMESTAMP '1992-09-20 11:30:00.123456' - interval_value as new_value from -- Interval shortened names SELECT INTERVAL '55h'; -+--------------------------------------------------------+ -| IntervalMonthDayNano("198000000000000") | -+--------------------------------------------------------+ -| 0 years 0 mons 0 days 55 hours 0 mins 0.000000000 secs | -+--------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 198000000000000 }") | ++---------------------------------------------------------------------------------------------------+ +| 55 hours | ++---------------------------------------------------------------------------------------------------+ SELECT INTERVAL '-2mon'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("-158456325028528675187087900672") | -+---------------------------------------------------------+ -| 0 years -2 mons 0 days 0 hours 0 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: -2, days: 0, nanoseconds: 0 }") | ++--------------------------------------------------------------------------------------+ +| -2 mons | ++--------------------------------------------------------------------------------------+ SELECT INTERVAL '-1h5m'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("18446740773709551616") | -+---------------------------------------------------------+ -| 0 years 0 mons 0 days 0 hours -55 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: -3300000000000 }") | ++--------------------------------------------------------------------------------------------------+ +| -55 mins | ++--------------------------------------------------------------------------------------------------+ SELECT INTERVAL '-1h-5m'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("18446740173709551616") | -+---------------------------------------------------------+ -| 0 years 0 mons 0 days -1 hours -5 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: -3900000000000 }") | ++--------------------------------------------------------------------------------------------------+ +| -1 hours -5 mins | ++--------------------------------------------------------------------------------------------------+ SELECT INTERVAL '1y2w3d4h'; -+---------------------------------------------------------+ -| IntervalMonthDayNano("950737950484766714775589781504") | -+---------------------------------------------------------+ -| 0 years 12 mons 17 days 4 hours 0 mins 0.000000000 secs | -+---------------------------------------------------------+ ++----------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 12, days: 17, nanoseconds: 14400000000000 }") | ++----------------------------------------------------------------------------------------------------+ +| 12 mons 17 days 4 hours | ++----------------------------------------------------------------------------------------------------+ SELECT '3y2mon'::INTERVAL; -+---------------------------------------------------------+ -| IntervalMonthDayNano("3010670175542044828554670112768") | -+---------------------------------------------------------+ -| 0 years 38 mons 0 days 0 hours 0 mins 0.000000000 secs | -+---------------------------------------------------------+ ++--------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 38, days: 0, nanoseconds: 0 }") | ++--------------------------------------------------------------------------------------+ +| 38 mons | ++--------------------------------------------------------------------------------------+ SELECT INTERVAL '7 days' - INTERVAL '1d'; -+----------------------------------------------------------------------------------------------+ -| IntervalMonthDayNano("129127208515966861312") - IntervalMonthDayNano("18446744073709551616") | -+----------------------------------------------------------------------------------------------+ -| 0 years 0 mons 6 days 0 hours 0 mins 0.000000000 secs | -+----------------------------------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 7, nanoseconds: 0 }") - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 1, nanoseconds: 0 }") | ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 6 days | ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ SELECT INTERVAL '2h' + INTERVAL '1h'; -+-------------------------------------------------------------------------------+ -| IntervalMonthDayNano("7200000000000") + IntervalMonthDayNano("3600000000000") | -+-------------------------------------------------------------------------------+ -| 0 years 0 mons 0 days 3 hours 0 mins 0.000000000 secs | -+-------------------------------------------------------------------------------+ ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 7200000000000 }") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 3600000000000 }") | ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 3 hours | ++---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -- Interval ISO 8601 SELECT INTERVAL 'p3y3m700dt133h17m36.789s'; -+-------------------------------------------------------------+ -| IntervalMonthDayNano("163343856789000000") | -+-------------------------------------------------------------+ -| 0 years 0 mons 0 days 45373 hours 17 mins 36.789000000 secs | -+-------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 163343856789000000 }") | ++------------------------------------------------------------------------------------------------------+ +| 45373 hours 17 mins 36.789000000 secs | ++------------------------------------------------------------------------------------------------------+ SELECT INTERVAL '-P3Y3M700DT133H17M36.789S'; -+----------------------------------------------------------------+ -| IntervalMonthDayNano("18283400216920551616") | -+----------------------------------------------------------------+ -| 0 years 0 mons 0 days -45373 hours -17 mins -36.789000000 secs | -+----------------------------------------------------------------+ ++-------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: -163343856789000000 }") | ++-------------------------------------------------------------------------------------------------------+ +| -45373 hours -17 mins -36.789000000 secs | ++-------------------------------------------------------------------------------------------------------+ SELECT 'P3Y3M700DT133H17M36.789S'::INTERVAL; -+-------------------------------------------------------------+ -| IntervalMonthDayNano("163343856789000000") | -+-------------------------------------------------------------+ -| 0 years 0 mons 0 days 45373 hours 17 mins 36.789000000 secs | -+-------------------------------------------------------------+ ++------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 163343856789000000 }") | ++------------------------------------------------------------------------------------------------------+ +| 45373 hours 17 mins 36.789000000 secs | ++------------------------------------------------------------------------------------------------------+ SELECT INTERVAL '2h' + INTERVAL 'P3Y3M700DT133H17M36.789S'; -+------------------------------------------------------------------------------------+ -| IntervalMonthDayNano("7200000000000") + IntervalMonthDayNano("163343856789000000") | -+------------------------------------------------------------------------------------+ -| 0 years 0 mons 0 days 45375 hours 17 mins 36.789000000 secs | -+------------------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 7200000000000 }") + IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 163343856789000000 }") | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| 45375 hours 17 mins 36.789000000 secs | ++--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -- Interval type does not support aggregation functions. -SELECT MIN(interval_value) from intervals; +SELECT min(interval_value) from intervals; -Error: 3001(EngineExecuteQuery), Internal error: Min/Max accumulator not implemented for type Interval(MonthDayNano). -This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker ++-------------------------------+ +| min(intervals.interval_value) | ++-------------------------------+ +| 12 mons | ++-------------------------------+ -SELECT MAX(interval_value) from intervals; +SELECT max(interval_value) from intervals; -Error: 3001(EngineExecuteQuery), Internal error: Min/Max accumulator not implemented for type Interval(MonthDayNano). -This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker ++-----------------------------------------------------+ +| max(intervals.interval_value) | ++-----------------------------------------------------+ +| 144 mons 12 days 12 hours 12 mins 12.012012012 secs | ++-----------------------------------------------------+ -SELECT SUM(interval_value) from intervals; +SELECT sum(interval_value) from intervals; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Execution("Sum not supported for Interval(MonthDayNano)") and No function matches the given name and argument types 'SUM(Interval(MonthDayNano))'. You might need to add explicit type casts. +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Execution("Sum not supported for Interval(MonthDayNano)") No function matches the given name and argument types 'sum(Interval(MonthDayNano))'. You might need to add explicit type casts. Candidate functions: - SUM(UserDefined) + sum(UserDefined) -SELECT AVG(interval_value) from intervals; +SELECT avg(interval_value) from intervals; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: No function matches the given name and argument types 'AVG(Interval(MonthDayNano))'. You might need to add explicit type casts. +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Plan("The function \"avg\" does not support inputs of type Interval(MonthDayNano).") No function matches the given name and argument types 'avg(Interval(MonthDayNano))'. You might need to add explicit type casts. Candidate functions: - AVG(Int8/Int16/Int32/Int64/UInt8/UInt16/UInt32/UInt64/Float32/Float64) + avg(UserDefined) DROP TABLE intervals; diff --git a/tests/cases/standalone/common/types/interval/interval.sql b/tests/cases/standalone/common/types/interval/interval.sql index 890d4ad202..08bdf48035 100644 --- a/tests/cases/standalone/common/types/interval/interval.sql +++ b/tests/cases/standalone/common/types/interval/interval.sql @@ -96,12 +96,12 @@ SELECT INTERVAL '2h' + INTERVAL 'P3Y3M700DT133H17M36.789S'; -- Interval type does not support aggregation functions. -SELECT MIN(interval_value) from intervals; +SELECT min(interval_value) from intervals; -SELECT MAX(interval_value) from intervals; +SELECT max(interval_value) from intervals; -SELECT SUM(interval_value) from intervals; +SELECT sum(interval_value) from intervals; -SELECT AVG(interval_value) from intervals; +SELECT avg(interval_value) from intervals; DROP TABLE intervals; diff --git a/tests/cases/standalone/common/types/string/scan_big_varchar.result b/tests/cases/standalone/common/types/string/scan_big_varchar.result index 2e26e72a50..d016ecc92a 100644 --- a/tests/cases/standalone/common/types/string/scan_big_varchar.result +++ b/tests/cases/standalone/common/types/string/scan_big_varchar.result @@ -34,7 +34,7 @@ Affected Rows: 1 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 1 | 1 | 10000 | 10000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -49,7 +49,7 @@ Affected Rows: 1 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 2 | 2 | 10000 | 20000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -61,7 +61,7 @@ Affected Rows: 2 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 4 | 4 | 10000 | 40000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -73,7 +73,7 @@ Affected Rows: 4 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 8 | 8 | 10000 | 80000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -85,7 +85,7 @@ Affected Rows: 8 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 16 | 16 | 10000 | 160000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -97,7 +97,7 @@ Affected Rows: 16 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 32 | 32 | 10000 | 320000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -109,7 +109,7 @@ Affected Rows: 32 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 64 | 64 | 10000 | 640000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -121,7 +121,7 @@ Affected Rows: 64 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 128 | 128 | 10000 | 1280000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -133,7 +133,7 @@ Affected Rows: 128 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 256 | 256 | 10000 | 2560000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -145,7 +145,7 @@ Affected Rows: 256 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 512 | 512 | 10000 | 5120000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -157,7 +157,7 @@ Affected Rows: 512 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 1024 | 1024 | 10000 | 10240000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -169,7 +169,7 @@ Affected Rows: 1024 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 2048 | 2048 | 10000 | 20480000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -181,7 +181,7 @@ Affected Rows: 2048 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 4096 | 4096 | 10000 | 40960000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -193,7 +193,7 @@ Affected Rows: 4096 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 8192 | 8192 | 10000 | 81920000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -202,7 +202,7 @@ SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 8192 | 8192 | 10000 | 81920000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -214,7 +214,7 @@ Affected Rows: 8192 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 16384 | 16384 | 10000 | 163840000 | +----------+-------------------+-----------------------------------+-----------------------------------+ @@ -226,7 +226,7 @@ Affected Rows: 16384 SELECT COUNT(*), COUNT(a), MAX(LENGTH(a)), SUM(LENGTH(a)) FROM bigtable; +----------+-------------------+-----------------------------------+-----------------------------------+ -| COUNT(*) | COUNT(bigtable.a) | MAX(character_length(bigtable.a)) | SUM(character_length(bigtable.a)) | +| count(*) | count(bigtable.a) | max(character_length(bigtable.a)) | sum(character_length(bigtable.a)) | +----------+-------------------+-----------------------------------+-----------------------------------+ | 32768 | 32768 | 10000 | 327680000 | +----------+-------------------+-----------------------------------+-----------------------------------+ diff --git a/tests/cases/standalone/common/types/string/unicode.result b/tests/cases/standalone/common/types/string/unicode.result index 903c4fe970..157d0a40ef 100644 --- a/tests/cases/standalone/common/types/string/unicode.result +++ b/tests/cases/standalone/common/types/string/unicode.result @@ -1,6 +1,6 @@ -- description: Test unicode strings -- insert unicode strings into the database -CREATE TABLE emojis(id INTEGER, s VARCHAR, ts timestamp time index); +CREATE TABLE emojis(`id` INTEGER, s VARCHAR, ts timestamp time index); Affected Rows: 0 diff --git a/tests/cases/standalone/common/types/string/unicode.sql b/tests/cases/standalone/common/types/string/unicode.sql index 3dae3e6ab2..b8f9656216 100644 --- a/tests/cases/standalone/common/types/string/unicode.sql +++ b/tests/cases/standalone/common/types/string/unicode.sql @@ -1,7 +1,7 @@ -- description: Test unicode strings -- insert unicode strings into the database -CREATE TABLE emojis(id INTEGER, s VARCHAR, ts timestamp time index); +CREATE TABLE emojis(`id` INTEGER, s VARCHAR, ts timestamp time index); INSERT INTO emojis VALUES (1, '🦆', 1), (2, '🦆🍞🦆', 2); diff --git a/tests/cases/standalone/common/types/timestamp/timestamp.result b/tests/cases/standalone/common/types/timestamp/timestamp.result index 0b799f7fe7..4c5398ae10 100644 --- a/tests/cases/standalone/common/types/timestamp/timestamp.result +++ b/tests/cases/standalone/common/types/timestamp/timestamp.result @@ -32,7 +32,7 @@ SELECT timestamp ' 2017-07-23 13:10:11 '; SELECT timestamp ' 2017-07-23 13:10:11 AA'; -Error: 3001(EngineExecuteQuery), Parser error: Invalid timezone "AA": 'AA' is not a valid timezone +Error: 3001(EngineExecuteQuery), Parser error: Invalid timezone "AA": failed to parse timezone SELECT timestamp 'AA2017-07-23 13:10:11'; @@ -60,7 +60,7 @@ SELECT t FROM timestamp ORDER BY t; SELECT MIN(t) FROM timestamp; +---------------------+ -| MIN(timestamp.t) | +| min(timestamp.t) | +---------------------+ | 2007-01-01T00:00:01 | +---------------------+ @@ -68,22 +68,22 @@ SELECT MIN(t) FROM timestamp; SELECT MAX(t) FROM timestamp; +---------------------+ -| MAX(timestamp.t) | +| max(timestamp.t) | +---------------------+ | 2008-02-01T00:00:01 | +---------------------+ SELECT SUM(t) FROM timestamp; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Execution("Sum not supported for Timestamp(Millisecond, None)") and No function matches the given name and argument types 'SUM(Timestamp(Millisecond, None))'. You might need to add explicit type casts. +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Execution("Sum not supported for Timestamp(Millisecond, None)") No function matches the given name and argument types 'sum(Timestamp(Millisecond, None))'. You might need to add explicit type casts. Candidate functions: - SUM(UserDefined) + sum(UserDefined) SELECT AVG(t) FROM timestamp; -Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: No function matches the given name and argument types 'AVG(Timestamp(Millisecond, None))'. You might need to add explicit type casts. +Error: 3000(PlanQuery), Failed to plan SQL: Error during planning: Execution error: User-defined coercion failed with Plan("The function \"avg\" does not support inputs of type Timestamp(Millisecond, None).") No function matches the given name and argument types 'avg(Timestamp(Millisecond, None))'. You might need to add explicit type casts. Candidate functions: - AVG(Int8/Int16/Int32/Int64/UInt8/UInt16/UInt32/UInt64/Float32/Float64) + avg(UserDefined) SELECT t+t FROM timestamp; @@ -121,7 +121,7 @@ SELECT EXTRACT(YEAR from TIMESTAMP '1992-01-01 01:01:01'); +-----------------------------------------------------+ | date_part(Utf8("YEAR"),Utf8("1992-01-01 01:01:01")) | +-----------------------------------------------------+ -| 1992.0 | +| 1992 | +-----------------------------------------------------+ SELECT EXTRACT(YEAR from TIMESTAMP '1992-01-01 01:01:01'::DATE); @@ -129,7 +129,7 @@ SELECT EXTRACT(YEAR from TIMESTAMP '1992-01-01 01:01:01'::DATE); +-----------------------------------------------------+ | date_part(Utf8("YEAR"),Utf8("1992-01-01 01:01:01")) | +-----------------------------------------------------+ -| 1992.0 | +| 1992 | +-----------------------------------------------------+ SELECT (TIMESTAMP '1992-01-01 01:01:01')::DATE; diff --git a/tests/cases/standalone/common/types/timestamp/timestamp_join.result b/tests/cases/standalone/common/types/timestamp/timestamp_join.result index 1b25cab706..9dd47893c9 100644 --- a/tests/cases/standalone/common/types/timestamp/timestamp_join.result +++ b/tests/cases/standalone/common/types/timestamp/timestamp_join.result @@ -17,7 +17,7 @@ Affected Rows: 1 select count(*) from timestamp2 inner join timestamp1 on (timestamp1.i = timestamp2.i); +----------+ -| COUNT(*) | +| count(*) | +----------+ | 1 | +----------+ diff --git a/tests/cases/standalone/common/types/timestamp/timestamp_precision.result b/tests/cases/standalone/common/types/timestamp/timestamp_precision.result index 0587aa72a6..58b91a92e0 100644 --- a/tests/cases/standalone/common/types/timestamp/timestamp_precision.result +++ b/tests/cases/standalone/common/types/timestamp/timestamp_precision.result @@ -1,6 +1,6 @@ -- description: Test timestamp precision CREATE TABLE ts_precision( - sec TIMESTAMP, + sec TIMESTAMP(0), msec TIMESTAMP(3), micros TIMESTAMP(6), nanos TIMESTAMP (9) TIME INDEX @@ -15,15 +15,19 @@ Affected Rows: 1 -- FIXME: nano seconds do not render correctly SELECT CAST(sec AS VARCHAR), CAST(msec AS VARCHAR), CAST(micros AS VARCHAR), CAST(nanos AS VARCHAR) FROM ts_precision; -+-------------------------+-------------------------+----------------------------+-------------------------------+ -| ts_precision.sec | ts_precision.msec | ts_precision.micros | ts_precision.nanos | -+-------------------------+-------------------------+----------------------------+-------------------------------+ -| 2020-01-01T01:23:45.123 | 2020-01-01T01:23:45.123 | 2020-01-01T01:23:45.123456 | 2020-01-01T01:23:45.123456789 | -+-------------------------+-------------------------+----------------------------+-------------------------------+ ++---------------------+-------------------------+----------------------------+-------------------------------+ +| ts_precision.sec | ts_precision.msec | ts_precision.micros | ts_precision.nanos | ++---------------------+-------------------------+----------------------------+-------------------------------+ +| 2020-01-01T01:23:45 | 2020-01-01T01:23:45.123 | 2020-01-01T01:23:45.123456 | 2020-01-01T01:23:45.123456789 | ++---------------------+-------------------------+----------------------------+-------------------------------+ SELECT EXTRACT(MICROSECONDS FROM sec), EXTRACT(MICROSECONDS FROM msec), EXTRACT(MICROSECONDS FROM micros), EXTRACT(MICROSECONDS FROM nanos) FROM ts_precision; -Error: 3001(EngineExecuteQuery), Execution error: Date part 'MICROSECONDS' not supported ++--------------------------------------------------+---------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+ +| date_part(Utf8("MICROSECONDS"),ts_precision.sec) | date_part(Utf8("MICROSECONDS"),ts_precision.msec) | date_part(Utf8("MICROSECONDS"),ts_precision.micros) | date_part(Utf8("MICROSECONDS"),ts_precision.nanos) | ++--------------------------------------------------+---------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+ +| 45000000 | 45123000 | 45123456 | 45123456 | ++--------------------------------------------------+---------------------------------------------------+-----------------------------------------------------+----------------------------------------------------+ -- we only support precisions 0, 3, 6, and 9 -- any other precision is rounded up (e.g. 1/2 -> 3, 4/5 -> 6, 7/8 -> 9) diff --git a/tests/cases/standalone/common/types/timestamp/timestamp_precision.sql b/tests/cases/standalone/common/types/timestamp/timestamp_precision.sql index 0bbda85794..d263037fa4 100644 --- a/tests/cases/standalone/common/types/timestamp/timestamp_precision.sql +++ b/tests/cases/standalone/common/types/timestamp/timestamp_precision.sql @@ -1,7 +1,7 @@ -- description: Test timestamp precision CREATE TABLE ts_precision( - sec TIMESTAMP, + sec TIMESTAMP(0), msec TIMESTAMP(3), micros TIMESTAMP(6), nanos TIMESTAMP (9) TIME INDEX diff --git a/tests/cases/standalone/common/types/timestamp/timestamp_types.result b/tests/cases/standalone/common/types/timestamp/timestamp_types.result index e134d932c6..0d453e376e 100644 --- a/tests/cases/standalone/common/types/timestamp/timestamp_types.result +++ b/tests/cases/standalone/common/types/timestamp/timestamp_types.result @@ -20,7 +20,7 @@ SELECT extract(YEAR from sec),extract( YEAR from milli),extract(YEAR from nano) +---------------------------------------+-----------------------------------------+----------------------------------------+ | date_part(Utf8("YEAR"),timestamp.sec) | date_part(Utf8("YEAR"),timestamp.milli) | date_part(Utf8("YEAR"),timestamp.nano) | +---------------------------------------+-----------------------------------------+----------------------------------------+ -| 2008.0 | 2008.0 | 2008.0 | +| 2008 | 2008 | 2008 | +---------------------------------------+-----------------------------------------+----------------------------------------+ SELECT nano::TIMESTAMP, milli::TIMESTAMP,sec::TIMESTAMP from timestamp; @@ -212,7 +212,7 @@ Affected Rows: 1 select count(*), nano from timestamp group by nano order by nano; +----------+-------------------------------+ -| COUNT(*) | nano | +| count(*) | nano | +----------+-------------------------------+ | 1 | 2008-01-01T00:00:01.889268321 | | 2 | 2008-01-01T00:00:01.899268321 | @@ -222,7 +222,7 @@ select count(*), nano from timestamp group by nano order by nano; select count(*), sec from timestamp group by sec order by sec; +----------+---------------------+ -| COUNT(*) | sec | +| count(*) | sec | +----------+---------------------+ | 1 | 2008-01-01T00:00:01 | | 2 | 2008-01-01T00:00:11 | @@ -232,7 +232,7 @@ select count(*), sec from timestamp group by sec order by sec; select count(*), milli from timestamp group by milli order by milli; +----------+-------------------------+ -| COUNT(*) | milli | +| count(*) | milli | +----------+-------------------------+ | 1 | 2008-01-01T00:00:01.594 | | 2 | 2008-01-01T00:00:01.794 | diff --git a/tests/cases/standalone/common/view/columns.result b/tests/cases/standalone/common/view/columns.result index 567b14c82c..7184cd3da1 100644 --- a/tests/cases/standalone/common/view/columns.result +++ b/tests/cases/standalone/common/view/columns.result @@ -210,25 +210,9 @@ ALTER TABLE t1 DROP COLUMN n; Affected Rows: 0 --- FIXME(dennis): The result looks weird, --- Looks like substrait referes to columns only by their relative indices, so that’s name-independent. --- Limit: skip=0, fetch=5 --- Projection: greptime.public.t1.ts, greptime.public.t1.s --- MergeScan [is_placeholder=false] --- Limit: skip=0, fetch=5 --- MergeScan [is_placeholder=false] --- See https://github.com/apache/datafusion/issues/6489 SELECT * FROM v1; -+-------------------------+---+ -| ts | s | -+-------------------------+---+ -| 1970-01-01T00:00:00.001 | | -| 1970-01-01T00:00:00.002 | | -| 1970-01-01T00:00:00.003 | | -| 1970-01-01T00:00:00.004 | | -| 1970-01-01T00:00:00.005 | | -+-------------------------+---+ +Error: 1002(Unexpected), Failed to decode DataFusion plan: No field named n. Valid fields are greptime.public.t1.ts, greptime.public.t1.s. DROP VIEW v1; diff --git a/tests/cases/standalone/common/view/columns.sql b/tests/cases/standalone/common/view/columns.sql index 2aee8d8c63..d556bd81f5 100644 --- a/tests/cases/standalone/common/view/columns.sql +++ b/tests/cases/standalone/common/view/columns.sql @@ -45,14 +45,6 @@ SELECT * FROM v1; ALTER TABLE t1 DROP COLUMN n; --- FIXME(dennis): The result looks weird, --- Looks like substrait referes to columns only by their relative indices, so that’s name-independent. --- Limit: skip=0, fetch=5 --- Projection: greptime.public.t1.ts, greptime.public.t1.s --- MergeScan [is_placeholder=false] --- Limit: skip=0, fetch=5 --- MergeScan [is_placeholder=false] --- See https://github.com/apache/datafusion/issues/6489 SELECT * FROM v1; DROP VIEW v1; diff --git a/tests/cases/standalone/common/view/create.result b/tests/cases/standalone/common/view/create.result index 06d892d189..e7651bb8ad 100644 --- a/tests/cases/standalone/common/view/create.result +++ b/tests/cases/standalone/common/view/create.result @@ -5,11 +5,11 @@ Affected Rows: 0 CREATE VIEW test_view; -Error: 2000(InvalidSyntax), sql parser error: Expected AS, found: ; at Line: 1, Column 22 +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Expected: AS, found: ; at Line: 1, Column: 22 CREATE VIEW test_view as DELETE FROM public.numbers; -Error: 2000(InvalidSyntax), sql parser error: Expected SELECT, VALUES, or a subquery in the query body, found: DELETE at Line: 1, Column 26 +Error: 2000(InvalidSyntax), Invalid SQL syntax: sql parser error: Expected: SELECT, VALUES, or a subquery in the query body, found: DELETE at Line: 1, Column: 26 --- Table already exists --- CREATE VIEW test_table as SELECT * FROM public.numbers; diff --git a/tests/cases/standalone/common/view/view.result b/tests/cases/standalone/common/view/view.result index 6e12202114..1ba9e9ec8f 100644 --- a/tests/cases/standalone/common/view/view.result +++ b/tests/cases/standalone/common/view/view.result @@ -16,17 +16,19 @@ Affected Rows: 0 SELECT * FROM v1; +-------------------------+ -| i | +| j | +-------------------------+ | 1970-01-01T00:00:00.041 | | 1970-01-01T00:00:00.042 | +-------------------------+ --- FIXME(dennis): Substrait doesn't support alias in projection -- --- https://github.com/apache/datafusion/issues/6489 -- SELECT j FROM v1 WHERE j > 41; -Error: 3000(PlanQuery), Failed to plan SQL: No field named j. Valid fields are v1.i. ++-------------------------+ +| j | ++-------------------------+ +| 1970-01-01T00:00:00.042 | ++-------------------------+ SELECT x FROM v1 t1(x) WHERE x > 41; @@ -36,33 +38,32 @@ SELECT x FROM v1 t1(x) WHERE x > 41; | 1970-01-01T00:00:00.042 | +-------------------------+ --- FIXME(dennis): DROP VIEW not supported yet-- --- DROP VIEW v1 -- --- substrait can't process such query currently --- CREATE VIEW v1 AS SELECT 'whatever';-- --- SELECT * FROM v1; -- --- substrait can't process such query currently ---CREATE OR REPLACE VIEW v1 AS SELECT 42;-- ---SELECT * FROM v1;-- INSERT INTO v1 VALUES (1); Error: 1004(InvalidArguments), Invalid SQL, error: column count mismatch, columns: 0, values: 1 +DROP VIEW v1; + +Affected Rows: 0 + +-- substrait can't process such query currently +-- CREATE VIEW v1 AS SELECT 'whatever'; +-- SELECT * FROM v1; +-- substrait can't process such query currently +-- CREATE OR REPLACE VIEW v1 AS SELECT 42; +-- SELECT * FROM v1; CREATE VIEW v1 AS SELECT * FROM dontexist; Error: 4001(TableNotFound), Failed to plan SQL: Table not found: greptime.public.dontexist SHOW VIEWS; -+-------+ -| Views | -+-------+ -| v1 | -+-------+ +++ +++ DROP VIEW v1; -Affected Rows: 0 +Error: 4001(TableNotFound), Table not found: greptime.public.v1 SELECT * FROM v1; diff --git a/tests/cases/standalone/common/view/view.sql b/tests/cases/standalone/common/view/view.sql index 9c5efc01b5..58a1e7391c 100644 --- a/tests/cases/standalone/common/view/view.sql +++ b/tests/cases/standalone/common/view/view.sql @@ -9,25 +9,23 @@ FROM t1 WHERE i < 43; SELECT * FROM v1; --- FIXME(dennis): Substrait doesn't support alias in projection -- --- https://github.com/apache/datafusion/issues/6489 -- SELECT j FROM v1 WHERE j > 41; SELECT x FROM v1 t1(x) WHERE x > 41; --- FIXME(dennis): DROP VIEW not supported yet-- --- DROP VIEW v1 -- - --- substrait can't process such query currently --- CREATE VIEW v1 AS SELECT 'whatever';-- --- SELECT * FROM v1; -- - --- substrait can't process such query currently ---CREATE OR REPLACE VIEW v1 AS SELECT 42;-- ---SELECT * FROM v1;-- - INSERT INTO v1 VALUES (1); +DROP VIEW v1; + +-- substrait can't process such query currently +-- CREATE VIEW v1 AS SELECT 'whatever'; +-- SELECT * FROM v1; + +-- substrait can't process such query currently +-- CREATE OR REPLACE VIEW v1 AS SELECT 42; +-- SELECT * FROM v1; + + CREATE VIEW v1 AS SELECT * FROM dontexist; SHOW VIEWS; diff --git a/tests/cases/standalone/optimizer/filter_push_down.result b/tests/cases/standalone/optimizer/filter_push_down.result index 06ee859d4a..33ce01865b 100644 --- a/tests/cases/standalone/optimizer/filter_push_down.result +++ b/tests/cases/standalone/optimizer/filter_push_down.result @@ -181,7 +181,7 @@ SELECT i FROM (SELECT * FROM integers i1 UNION SELECT * FROM integers i2) a WHER SELECT * FROM (SELECT i1.i AS a, i2.i AS b, row_number() OVER (ORDER BY i1.i, i2.i) FROM integers i1, integers i2 WHERE i1.i IS NOT NULL AND i2.i IS NOT NULL) a1 WHERE a=b ORDER BY 1; +---+---+--------------------------------------------------------------------------------------------------------------------+ -| a | b | ROW_NUMBER() ORDER BY [i1.i ASC NULLS LAST, i2.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +| a | b | row_number() ORDER BY [i1.i ASC NULLS LAST, i2.i ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW | +---+---+--------------------------------------------------------------------------------------------------------------------+ | 1 | 1 | 1 | | 2 | 2 | 5 | @@ -190,9 +190,15 @@ SELECT * FROM (SELECT i1.i AS a, i2.i AS b, row_number() OVER (ORDER BY i1.i, i2 -- TODO(ruihang): Invalid argument error: must either specify a row count or at least one column -- SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2) a1 WHERE cond ORDER BY 1; +-- This should be a bug in DataFusion, it use UserDefinedLogicalNode::prevent_predicate_push_down_columns() +-- to prevent pushdown, but this filter is Literal(Boolean(false)). It doesn't reference any column. SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2 GROUP BY 1) a1 WHERE cond ORDER BY 1; -Error: 3001(EngineExecuteQuery), DataFusion error: Error during planning: Attempted to create Filter predicate with expression `Boolean(false)` aliased as 'Int64(0) = Int64(1)'. Filter predicates should not be aliased. ++-------+ +| cond | ++-------+ +| false | ++-------+ DROP TABLE integers; diff --git a/tests/cases/standalone/optimizer/filter_push_down.sql b/tests/cases/standalone/optimizer/filter_push_down.sql index b2cd11582d..0d47ed3713 100644 --- a/tests/cases/standalone/optimizer/filter_push_down.sql +++ b/tests/cases/standalone/optimizer/filter_push_down.sql @@ -52,6 +52,8 @@ SELECT * FROM (SELECT i1.i AS a, i2.i AS b, row_number() OVER (ORDER BY i1.i, i2 -- TODO(ruihang): Invalid argument error: must either specify a row count or at least one column -- SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2) a1 WHERE cond ORDER BY 1; +-- This should be a bug in DataFusion, it use UserDefinedLogicalNode::prevent_predicate_push_down_columns() +-- to prevent pushdown, but this filter is Literal(Boolean(false)). It doesn't reference any column. SELECT * FROM (SELECT 0=1 AS cond FROM integers i1, integers i2 GROUP BY 1) a1 WHERE cond ORDER BY 1; DROP TABLE integers; diff --git a/tests/cases/standalone/optimizer/last_value.result b/tests/cases/standalone/optimizer/last_value.result index 790a6a4748..fe1bf50305 100644 --- a/tests/cases/standalone/optimizer/last_value.result +++ b/tests/cases/standalone/optimizer/last_value.result @@ -45,7 +45,6 @@ explain analyze |_|_|_AggregateExec: mode=FinalPartitioned, gby=[host@0 as host], aggr=[last_value(t.host) ORDER BY [t.ts ASC NULLS LAST], last_value(t.not_pk) ORDER BY [t.ts ASC NULLS LAST], last_value(t.val) ORDER BY [t.ts ASC NULLS LAST]] REDACTED |_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_RepartitionExec: REDACTED -|_|_|_CoalesceBatchesExec: target_batch_size=8192 REDACTED |_|_|_AggregateExec: mode=Partial, gby=[host@1 as host], aggr=[last_value(t.host) ORDER BY [t.ts ASC NULLS LAST], last_value(t.not_pk) ORDER BY [t.ts ASC NULLS LAST], last_value(t.val) ORDER BY [t.ts ASC NULLS LAST]] REDACTED |_|_|_RepartitionExec: REDACTED |_|_|_SeqScan: region=REDACTED, partition_count=1 (1 memtable ranges, 0 file 0 ranges), selector=LastRow REDACTED diff --git a/tests/cases/standalone/optimizer/order_by.result b/tests/cases/standalone/optimizer/order_by.result index e7fd0c073d..038e47bbfd 100644 --- a/tests/cases/standalone/optimizer/order_by.result +++ b/tests/cases/standalone/optimizer/order_by.result @@ -32,25 +32,23 @@ explain select * from numbers order by number asc; explain select * from numbers order by number desc limit 10; -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| plan_type | plan | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| logical_plan | MergeScan [is_placeholder=false] | -| physical_plan | GlobalLimitExec: skip=0, fetch=10 | -| | SortExec: TopK(fetch=10), expr=[number@0 DESC], preserve_partitioning=[false] | -| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | -| | | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| plan_type | plan | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| logical_plan | MergeScan [is_placeholder=false] | +| physical_plan | SortExec: TopK(fetch=10), expr=[number@0 DESC], preserve_partitioning=[false] | +| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | +| | | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ explain select * from numbers order by number asc limit 10; -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| plan_type | plan | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| logical_plan | MergeScan [is_placeholder=false] | -| physical_plan | GlobalLimitExec: skip=0, fetch=10 | -| | SortExec: TopK(fetch=10), expr=[number@0 ASC NULLS LAST], preserve_partitioning=[false] | -| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | -| | | -+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| plan_type | plan | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| logical_plan | MergeScan [is_placeholder=false] | +| physical_plan | SortExec: TopK(fetch=10), expr=[number@0 ASC NULLS LAST], preserve_partitioning=[false] | +| | StreamScanAdapter: [], schema: [Schema { fields: [Field { name: "number", data_type: UInt32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }], metadata: {"greptime:version": "0"} }] | +| | | ++---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+