Ruihang Xia
0479c05092
perf: preserve dictionary-encoded query labels ( #8541 )
...
* perf: preserve dictionary-encoded query labels
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(client): skip dictionary Flight batches
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(servers): decode dictionary labels in HTTP output
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* test(mito2): support dictionary tags in series scans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(servers): preserve dictionary child nulls in SQL
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(promql): compare dictionary tags by logical nulls
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): keep dictionary tags within query paths
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refactor(query): scope PK dictionary encoding to reads
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): preserve pushdown for dictionary labels
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): handle dictionary labels in query operators
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): complete label type matching
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): align dictionary query schemas
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): decode incompatible OR labels
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix(query): preserve dictionary partition pruning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: handle dictionary query edge cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: import dictionary downcast macro
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2026-07-17 13:57:10 +00:00
jeremyhi
5bcc2e9d6b
fix: global limit for distributed inspect streams ( #8412 )
...
* fix: global limit for distributed inspect streams
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
* fix: honor tighter filter fetch under global limit
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
* fix: restore distribution after global limit
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
* fix: preserve inherited distribution for global limit
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
* fix: propagate inherited hash partitioning
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com >
2026-07-06 07:02:54 +00:00
discord9
4f5dccf6bb
fix(query): push down PromQL cast filters ( #8407 )
...
* fix(query): push down PromQL cast filters
Signed-off-by: discord9 <discord9@163.com >
* test(query): cover cast preimage pushdown
Signed-off-by: discord9 <discord9@163.com >
* test(query): expand cast preimage coverage
Signed-off-by: discord9 <discord9@163.com >
* test(query): compare cast normalization rules
Signed-off-by: discord9 <discord9@163.com >
* test(query): update cast pushdown sqlness plans
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2026-07-03 09:10:00 +00:00
discord9
9cf071808a
fix(query): run optimizer rules before MergeScan ( #8339 )
...
* fix(query): push down join filters before MergeScan
Signed-off-by: discord9 <discord9@163.com >
* fix(query): run optimizer before MergeScan pushdown
Signed-off-by: discord9 <discord9@163.com >
* fix(query): narrow pre-MergeScan filter pushdown
Signed-off-by: discord9 <discord9@163.com >
* fix(query): refine pre-MergeScan optimizer prepass
Signed-off-by: discord9 <discord9@163.com >
* fix(query): satisfy predicate extractor clippy
Signed-off-by: discord9 <discord9@163.com >
* test(query): cover pre-MergeScan optimizer edges
Signed-off-by: discord9 <discord9@163.com >
* test(query): cover set comparison prepass
Signed-off-by: discord9 <discord9@163.com >
* fix(query): guard remote scan filter pushdown
Signed-off-by: discord9 <discord9@163.com >
* fix(query): preserve subquery planning errors
Signed-off-by: discord9 <discord9@163.com >
* fix(query): preserve usable scan predicates
Signed-off-by: discord9 <discord9@163.com >
* fix(query): simplify scan predicate extraction
Signed-off-by: discord9 <discord9@163.com >
* fix(query): keep scan filter extraction scoped
Signed-off-by: discord9 <discord9@163.com >
* docs(query): explain pre-MergeScan optimizer
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2026-06-23 12:15:40 +00:00
Ruihang Xia
85ae29cb0c
perf: collect narrow binary join ( #8193 )
...
* perf(promql): collect narrow binary join build side
* fix projection
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finalize
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename mod
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2026-05-28 11:30:30 +00:00
Ruihang Xia
9133d0464f
feat: pre-cast constants ( #7926 )
...
* init impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle no cast
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refactor using common-expr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* extend matching pattern
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplification
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix zero timestamp
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: normalize sqlness partition count output
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: normalize remaining sqlness plan output
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: normalize sqlness repartition details in tql explain
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: tighten const normalization casts
* test: normalize standalone tql explain repartition output
* resolve cr comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2026-05-12 09:23:43 +00:00
LFC
7a285c2890
feat: concretize json type from query ( #8081 )
...
* feat: concretize json type from query
Signed-off-by: luofucong <luofc@foxmail.com >
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com >
* add more tests
Signed-off-by: luofucong <luofc@foxmail.com >
---------
Signed-off-by: luofucong <luofc@foxmail.com >
2026-05-09 07:27:40 +00:00
Ning Sun and Ruihang Xia
9b1288f66a
feat: add function rewrite rule for json_get with cast ( #7631 )
...
* feat: initial function rewriter for json_get
* feat: make sure rewrite rule is applied
* feat: keep analyzer's default rules
* feat: implement rewriter for arrow_cast
* test: add unit test for tht rewriter
* chore: format
* refactor: extract some more functions
* Apply suggestion from @waynexia
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2026-03-09 02:39:48 +00:00
b2074e3863
chore: upgrade DataFusion family, again ( #7578 )
...
* chore: upgrade DataFusion family
Signed-off-by: luofucong <luofc@foxmail.com >
* chore: switch to released version of datafusion-pg-catalog
---------
Signed-off-by: luofucong <luofc@foxmail.com >
Co-authored-by: Ning Sun <sunning@greptime.com >
Co-authored-by: Ning Sun <sunng@protonmail.com >
2026-03-03 07:36:39 +00:00
fys
1aa80d9363
fix: incorrect-tql-explain result ( #7675 )
2026-02-11 02:30:15 +00:00
Ruihang Xia
4ac73a7e9e
feat: adapt prometheus 3.x selector and lookback behavior ( #7688 )
...
* feat: adapt prometheus 3.x selector and lookback behavior
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove commit hash from comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2026-02-10 04:00:45 +00:00
LFC and Ning Sun
2f637a262e
chore: update datafusion to 50 ( #7076 )
...
* chore: update datafusion to 50
Signed-off-by: luofucong <luofc@foxmail.com >
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com >
* fix: update datafusion_pg_catalog import
* chore: fix toml format
* chore: fix toml format again
* fix nextest
Signed-off-by: luofucong <luofc@foxmail.com >
* fix sqlness
Signed-off-by: luofucong <luofc@foxmail.com >
* chore: switch datafusion-orc to upstream tag
* fix sqlness
Signed-off-by: luofucong <luofc@foxmail.com >
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com >
---------
Signed-off-by: luofucong <luofc@foxmail.com >
Co-authored-by: Ning Sun <sunning@greptime.com >
2025-10-23 07:18:36 +00:00
dennis zhuang
57d84b9de5
feat: supports value aliasing in TQL ( #7041 )
...
* feat: supports value aliasing in TQL
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* fix: invalid checking
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* chore: remove invalid checking
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* test: add explain test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* chore: improve parser
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* test: add explain TQL-CTE
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
2025-10-11 02:49:09 +00:00
discord9
aca8b690d1
fix: step aggr merge phase not order nor filter ( #6998 )
...
* fix: not order
Signed-off-by: discord9 <discord9@163.com >
* test: redacted
Signed-off-by: discord9 <discord9@163.com >
* feat: fix up state wrapper
Signed-off-by: discord9 <discord9@163.com >
* df last_value state not as promised!
Signed-off-by: discord9 <discord9@163.com >
* fix?: could fix better
Signed-off-by: discord9 <discord9@163.com >
* test: unstable result
Signed-off-by: discord9 <discord9@163.com >
* fix: work around by fixing state
Signed-off-by: discord9 <discord9@163.com >
* chore: after rebase fix
Signed-off-by: discord9 <discord9@163.com >
* chore: finish some todo
Signed-off-by: discord9 <discord9@163.com >
* chore: per copilot
Signed-off-by: discord9 <discord9@163.com >
* refactor: not fix but just notify mismatch
Signed-off-by: discord9 <discord9@163.com >
* chore: warn -> debug state mismatch
Signed-off-by: discord9 <discord9@163.com >
* chore: refine error msg
Signed-off-by: discord9 <discord9@163.com >
* test: sqlness add last_value date_bin test
Signed-off-by: discord9 <discord9@163.com >
* ?: substrait order by decode failure
Signed-off-by: discord9 <discord9@163.com >
* unit test reproduce that
Signed-off-by: discord9 <discord9@163.com >
* feat: support state wrapper's order serde in substrait
Signed-off-by: discord9 <discord9@163.com >
* refactor: stuff
Signed-off-by: discord9 <discord9@163.com >
* test: standalone/distributed different exec
Signed-off-by: discord9 <discord9@163.com >
* fmt
Signed-off-by: discord9 <discord9@163.com >
* chore: per review
Signed-off-by: discord9 <discord9@163.com >
* refactor: closure
Signed-off-by: discord9 <discord9@163.com >
* test: first value order by
Signed-off-by: discord9 <discord9@163.com >
* refactor: per cr
Signed-off-by: discord9 <discord9@163.com >
* feat: ScanHint last_value last row selector
Signed-off-by: discord9 <discord9@163.com >
* docs: per cr
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-09-26 09:12:45 +00:00
Ruihang Xia
288f69a30f
fix: plan disorder from upgrading datafusion ( #6787 )
...
* fix: plan disorder from upgrading datafusion
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-08-23 12:29:47 +00:00
LFC
f9d2a89a0c
chore: update datafusion family ( #6675 )
...
* chore: update datafusion family
Signed-off-by: luofucong <luofc@foxmail.com >
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com >
* use official otel-arrow-rust
Signed-off-by: luofucong <luofc@foxmail.com >
* rebase
Signed-off-by: luofucong <luofc@foxmail.com >
* use the official orc-rust
Signed-off-by: luofucong <luofc@foxmail.com >
* resolve PR comments
Signed-off-by: luofucong <luofc@foxmail.com >
* remove the empty lines
Signed-off-by: luofucong <luofc@foxmail.com >
* try following PR comments
Signed-off-by: luofucong <luofc@foxmail.com >
---------
Signed-off-by: luofucong <luofc@foxmail.com >
2025-08-15 12:41:49 +00:00
discord9
6501b0b13c
feat: MergeScan print input ( #6563 )
...
* feat: MergeScan print input
Signed-off-by: discord9 <discord9@163.com >
* test: fix ut
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-07-21 06:40:15 +00:00
Ruihang Xia
bbb6f8685e
feat: implement commutativity rule for prom-related plans ( #5875 )
...
* feat: implement commutativity rule for prom-related plans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix range manipulate deserializer
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* blocklist in commutativity rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change dictionary type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle partition and ordering
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add rate, increase and delta
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* regexp_replace uses empty string instead of null value
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-05-13 09:06:25 +00:00
Ruihang Xia
b476584f56
feat: remove hyper parameter from promql functions ( #5955 )
...
* quantile udaf
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* extrapolate rate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* predict_linear, round, holt_winters, quantile_overtime
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix quantile function
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-24 07:17:10 +00:00
Ruihang Xia
dcf1a486f6
feat: support @@ (AtAt) operator for term matching ( #5902 )
...
* update dep and sqlness case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement transcribe rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-15 11:05:17 +00:00
Zhenchi
2189631efd
feat: optimize matches_term with constant term pre-compilation ( #5886 )
...
* feat: precompile finder for `matches_term`
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix sqlness
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2025-04-15 06:45:56 +00:00
Ruihang Xia
08d0f31865
perf: avoid duplicate computation in promql ( #5863 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-09 11:53:25 +00:00
Ruihang Xia
c16bae32c4
perf: evolve promql execution engine ( #5691 )
...
* use the same sort option across every prom plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tweak plans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wip
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix merge compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Revert "wip"
This reverts commit db58884236waynestxia@gmail.com >
* handle error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* pass distribution rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reverse sort order
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refine plans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more optimizations for plans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* check logical table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wierd tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add test for series_divide
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix scalar calculation
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: workaround join partition
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update proto
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-08 08:12:15 +00:00
Yingwen
dbc25dd8da
feat: expose scanner metrics to df execution metrics ( #5699 )
...
* feat: add metrics list to scanner
* chore: add report metrics method
* feat: use df metrics in PartitionMetrics
* feat: pass execution metrics to scan partition
* refactor: remove PartitionMetricsList
* feat: better debug format for ScanMetricsSet
* feat: do not expose all metrics to execution metrics by default
* refactor: use struct destruction
* feat: add metrics list to scanner
* chore: Add custom Debug for ScanMetricsSet and partition metrics display
* test: update sqlness result
2025-03-27 23:40:39 +00:00
Yingwen
d88482b996
feat: support explain analyze verbose ( #5763 )
...
* Add explain_verbose to QueryContext
* feat: fmt plan by display type
* feat: update proto to use ExplainOptions
* feat: display more info in verbose mode
* chore: fix clippy
* test: add sqlness test
* test: update sqlness result
* chore: update proto version
* chore: Simplify QueryContextBuilder::explain_options using get_or_insert_default
2025-03-25 03:48:36 +00:00
Ruihang Xia and evenyag
73ca39f37e
feat: time series distribution in scanner ( #5675 )
...
* define distribution
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: SeqScan support per series distribution
* probe distribution
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reverse sort order
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* more strict check
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change null's ordering
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
2025-03-10 05:43:17 +00:00
Yingwen and Ruihang Xia
9860bca986
feat: support exact filter on time index column ( #5671 )
...
* feat: add predicate group
* feat: pass predicate group
* feat: memtable prune by time filters
* test: test PruneTimeIterator with time filters
* feat: push down returns exact for timestamp simple filters
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2025-03-07 21:55:46 +00:00
Ruihang Xia
e463942a5b
fix: recover plan schema after dist analyzer ( #5665 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-03-07 00:29:55 +00:00
35b635f639
feat!: Bump datafusion, prost, hyper, tonic, tower, axum ( #5417 )
...
* change dep
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: adapt to arrow's interval array
* chore: fix compile errors in datatypes crate
* chore: fix api crate compiler errors
* chore: fix compiler errors in common-grpc
* chore: fix common-datasource errors
* chore: fix deprecated code in common-datasource
* fix promql and physical plan related
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wip: upgrading network deps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* block on updating `sqlparser`
* upgrade sqlparser
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* adapt new df's trait requirements
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: fix compiler errors in mito2
* chore: fix common-function crate errors
* chore: fix catalog errors
* change import path
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: fix some errors in query crate
* chore: fix some errors in query crate
* aggr expr and some other tiny fixes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: fix expr related errors in query crate
* chore: fix query serializer and admin command
* chore: fix grpc services
* feat: axum serve
* chore: fix http server
* remove handle_error handler
* refactor timeout layer
* serve axum
* chore: fix flow aggr functions
* chore: fix flow
* feat: fix errors in meta-srv
* boxed()
* use TokioIo
* feat!: Remove script crate and python feature (#5321 )
* feat: exclude script crate
* chore: simplify feature
* feat: remove the script crate
* chore: remove python feature and some comments
* chore: fix warning
* chore: fix servers tests compiler errors
* feat: fix tests-integration errors
* chore: fix unused
* test: fix catalog test
* chore: fix compiler errors for crates using common-meta
testing feature is enabled when check with --workspace
* test: use display for logical plan test
* test: implement rewrite for ScanHintRule
* fix: http server build panic
* test: fix mito test
* fix: sql parser type alias error
* test: fix TestClient not listen
* test: some flow tests
* test(flow): more fix
* fix: test_otlp_logs
* test: fix promql test that using deprecated method fun()
* fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64
* test: fix infer schema test case
* test: fix tests related to plan display
* chore: fix last flow test
* test: fix function format related assertion
* test: use larger port range for tests
* fix: test_otlp_traces
* fix: test_otlp_metrics
* fix range query and dist plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: flow handle distinct use deprecated field
* fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs
* test: fix deserialize test
* test: reduce split key case num
* tests: lower case aggr func name
* test: fix some sqlness tests
* tests: more sqlness fix
* tests: fixed sqlness test
* commit non-bug changes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: make our udf correct
* fix: implement empty methods of ContextProvider for DfContextProviderAdapter
* test: update sqlness test result
* chore: remove unused
* fix: provide alias name for AggregateExprBuilder in range plan
* test: update range query result
* fix: implement missing ContextProvider methods for DfContextProviderAdapter
* test: update timestamps, cte result
* fix: supports empty projection in mito
* test: update comment for cte test
* fix: support projection for numbers
* test: update test cases after projection fix
* fix: fix range select first_value/last_value
* fix: handle CAST and time index conflict
* fix: handle order by correctly in range first_value/last_value
* test: update sqlness result
* test: update view test result
* test: update decimal test
wait for https://github.com/apache/datafusion/pull/14126 to fix this
* feat: remove redundant physical optimization
todo(ruihang): Check if we can remove this.
* test: update sqlness test result
* chore: range select default sort use nulls_first = false
* test: update filter push down test result
* test: comment deciaml test to avoid different panic message
* test: update some distributed test result
* test: update test for distributed count and filter push down
* test: update subqueries test
* fix: SessionState may overwrite our UDFs
* chore: fix compiler errors after merging main
* fix: fix elasticsearch and dashboard router panic
* chore: fix common-functions tests
* chore: update sqlness result
* test: fix id keyword and update sqlness result
* test: fix flow_null test
* fix: enlarge thread size in debug mode to avoid overflow
* chore: fix warnings in common-function
* chore: fix warning in flow
* chore: fix warnings in query crate
* chore: remove unused warnings
* chore: fix deprecated warnings for parquet
* chore: fix deprecated warning in servers crate
* style: fix clippy
* test: enlarge mito cache tttl test ttl time
* chore: fix typo
* style: fmt toml
* refactor: reimplement PartialOrd for RangeSelect
* chore: remove script crate files introduced by merge
* fix: return error if sql option is not kv
* chore: do not use ..default::default()
* chore: per review
* chore: update error message in BuildAdminFunctionArgsSnafu
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
* refactor: typed precision
* update sqlness view case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: flow per review
* chore: add example in comment
* chore: warn if parquet stats of timestamp is not INT64
* style: add a newline before derive to make the comment more clear
* test: update sqlness result
* fix: flow from substrait
* chore: change update_range_context log to debug level
* chore: move axum-extra axum-macros to workspace
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: luofucong <luofc@foxmail.com >
Co-authored-by: discord9 <discord9@163.com >
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
2025-01-23 06:15:40 +00:00
Ruihang Xia
f9ebb58a12
fix: put PipelineChecker at the end ( #5092 )
...
fix: put PipelineChecker in the end
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-12-06 02:10:17 +00:00
03f2fa219d
feat: optimizer rule for windowed sort ( #4874 )
...
* basic impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement physical rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: install windowed sort physical rule and optimize partition ranges
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add logs and sqlness test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: introduce PartSortExec for partitioned sorting
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tune exec nodes' properties and metrics
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* debug: add more info on very wrong
* debug: also print overlap ranges
* feat: add check when emit PartSort Stream
* dbg: info on overlap working range
* feat: check batch range is inside part range
* set distinguish partition range param
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: more logs
* update sqlness
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tune optimizer
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix lints
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix windowed sort rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: early terminate sort stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: remove min/max check
* chore: remove unused windowed_sort module, uuid feature and refactor region_scanner to synchronous
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: print more fuzz log
* chore: more log
* fix: part sort should skip empty part
* chore: remove insert logs
* tests: empty PartitionRange
* refactor: testcase
* docs: update comment&tests: all empty
* ci: enlarge etcd cpu limit
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: discord9 <discord9@163.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
2024-10-29 07:46:05 +00:00
Yingwen and Ruihang Xia
3e17c09e45
feat: skip caching uncompressed pages if they are large ( #4705 )
...
* feat: cache each uncompressed page
* chore: remove unused function
* chore: log
* chore: log
* chore: row group pages cache kv
* feat: also support row group level cache
* chore: fix range count
* feat: don't cache compressed page for row group cache
* feat: use function to get part
* chore: log whether scan is from compaction
* chore: avoid get column
* feat: add timer metrics
* chore: Revert "feat: add timer metrics"
This reverts commit 4618f57fa2ba13b1e1a8dec83afd01c00ae4c867.
* feat: don't cache individual uncompressed page
* feat: append in row group level under append mode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: fetch pages cost
* perf: yield
* Update src/mito2/src/sst/parquet/row_group.rs
* refactor: cache key
* feat: print file num and row groups num in explain
* test: update sqlness test
* chore: Update src/mito2/src/sst/parquet/page_reader.rs
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-10 11:52:16 +00:00
Yingwen and dennis zhuang
f382a7695f
perf: reduce lock scope and improve log ( #4453 )
...
* feat: refine logs for scan
* feat: improve build parts and unordered scan metrics
* feat: change to debug log
* fix: release lock before reading part
* test: replace region id
* test: fix sqlness
* chore: add todo
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-07-31 04:07:34 +00:00
Ruihang Xia and Jeremyhi
16075ada67
feat: impl optimizer rule to handle last_value case ( #4357 )
...
* feat: impl optimizer rule to handle last_value case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/query/src/optimizer/scan_hint.rs
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
* split methods
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
2024-07-12 14:36:23 +00:00
Ruihang Xia
a7aa556763
feat: output multiple partition in MergeScanExec ( #4227 )
...
* feat: output multiple partition in MergeScanExec
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix range manipulate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-28 13:45:22 +00:00
Ruihang Xia
fce65c97e3
feat: make RegionScanner aware of PartitionRange ( #4170 )
...
* define PartitionRange
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add optimizer rule
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement interfaces
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl aggr stream
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add fallback method
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add document and rename struct
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add more comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-21 09:54:22 +00:00
Yingwen
fe74efdafe
feat: Implement memtable range ( #4162 )
...
* refactor: RangeBase
* feat: memtable range
* feat: scanner use mem range
* feat: remove base from mem range context
* feat: impl ranges for memtables
* chore: fix warnings
* refactor: make predicate cheap to clone
* refactor: MemRange -> MemtableRange
* feat: pub empty memtable to fix warnings
* test: fix sqlness result
2024-06-18 22:25:19 +00:00
Yingwen and Ruihang Xia
65f8b72d34
feat: Implement RegionScanner for SeqScan ( #4060 )
...
* feat: ordered builder wip
* feat: impl RegionScanner for SeqScan
* feat: implement scan_partition and build_stream
* chore: return SeqScan as RegionScanner
* fix: group parts
* feat: split parts
* chore: reader metrics
* chore: metrics
* chore: remove unused codes
* chore: support holding a group of ranges in ScanPart
* feat: group ScanParts to ScanParts
* feat: impl SeqScanner again
* chore: observe build cost in ScannerMetrics
* chore: fix compiler warnings
* style: fix clippy
* docs: update config docs
* chore: forward DisplayAs to scanner
* test: update sqlness tests
* chore: update debug fmt
* chore: custom debug for timestamp
fix test compiling issue with common-macro when running
cargo nextest -p common-time
* chore: update debug format
* feat: update fmt for scan part
* chore: fix warning
* fix: sanitize parallelism
* feat: split parts
* test: fix config api test
* feat: update logs
* chore: Revert "chore: remove unused codes"
This reverts commit b548b30a01eeded59b1a0a8d89f9293ca63afc41.
* chore: Revert "docs: update config docs"
This reverts commit a7997e78d6ddcf635560574de8c1948c495bdd12.
* feat: each partition scan files in parallel
* test: fix config api test
* docs: fix typo
* chore: address comments, simplify tests
* feat: global semaphore
* feat: always spawn task
* chore: simplify default explain output format
* handle output partiton number is 0
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-12 08:21:30 +00:00
Ruihang Xia and Jeremyhi
c2218f8be8
build(deps): bump datafusion 20240528 ( #4061 )
...
* build(deps): bump datafusion 20240528
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* another update
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update expected sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix first/last value
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* reformat comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix remaining errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* revert toml format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 feature
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove dead code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
* format file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
2024-06-01 14:03:00 +00:00
Yingwen
179c8c716c
feat: Adds RegionScanner trait ( #3948 )
...
* feat: define region scanner
* feat: single partition scanner
* feat: use single partition scanner
* feat: implement ExecutionPlan wip
* feat: mito engine returns single partition scanner
* feat: implement DisplayAs for region server
* feat: dummy table provider use handle_partitioned_query()
* test: update sqlness test
* feat: table provider use ReadFromRegion
* refactor: remove StreamScanAdapter
* chore: update lock
* style: fix clippy
* refactor: remove handle_query from the RegionEngine trait
* chore: address CR comments
* refactor: rename methods
* refactor: rename ReadFromRegion to RegionScanExec
2024-05-20 11:52:00 +00:00
Ruihang Xia and Jeremyhi
aec5cca2c7
feat: support distributed EXPLAIN ANALYZE ( #3908 )
...
* feat: fetch and pass per-plan metrics
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl DistAnalyzeExec
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness results
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo again
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/query/src/analyze.rs
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
2024-05-10 12:31:29 +00:00
Ruihang Xia
e84b1eefdf
perf: optimize count(*) ( #3845 )
...
* perf: optimize count(*)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fallback to count(1) for temporary table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle alias expr in range plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle subquery alias
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-30 10:10:11 +00:00
Ruihang Xia
a0f4881c6e
feat: physical optimizer RemoveDuplicate to remove duplicate exec plans ( #3839 )
...
* feat: physical optimizer RemoveDuplicate to remove duplicate exec plans
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update document
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness results
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-30 03:13:39 +00:00
314f2704d4
build(deps): update datafusion to latest and arrow to 51.0 ( #3661 )
...
* chore: update datafusion
* update sqlness case of time.sql
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: adjust range query partition
* fix: hisogram incorrect result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: ignore filter pushdown temporarily
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: update limit sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: histogram with wrong distribution
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: update negative ordinal sqlness case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* feat: bump df to cd7a00b
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve conflicts
* ignore test_range_filter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix promql exec panic
* fix "select count(*)" exec error
* re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed
* fix: range query schema error
* update sqlness results
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* resolve conflicts
* update datafusion, again
* fix pyo3 compile error, and update some sqlness results
* update decimal sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: promql literal
* fix udaf tests
* fix filter pushdown sqlness tests
* fix?: test_cast
* fix: rspy test fail due to datafusion `sin` signature change
* rebase main to see if there are any failed tests
* debug ci
* debug ci
* debug ci
* enforce input partition
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* debug ci
* fix ci
* fix ci
* debug ci
* debug ci
* debug ci
* fix sqlness
* feat: do not return error while creating a filter
* chore: remove array from error
* chore: replace todo with unimplemented
* Update src/flow/clippy.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: WUJingdi <taylor-lagrange@qq.com >
Co-authored-by: discord9 <discord9@163.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-18 12:07:18 +00:00
Eugene Tolbakov
bbea651d08
feat(promql): parameterize lookback ( #3630 )
...
* feat(promql): parameterize lookback
* chore(promql): address CR, adjusted sqlness
* chore(promql): fmt
* chore(promql): fix accidental removal
* fix(promql): address CR
* fix(promql): address CR
* feat(promql): add initial lookback parameter grpc support
* fix: update greptime-proto revision
* chore: restore accidental removal
2024-04-15 09:11:21 +00:00
Eugene Tolbakov
14267c2aed
feat(tql): add initial support for start,stop,step as sql functions ( #3507 )
...
* feat(tql): add initial support for start,stop,step as sql functions
* fix(tql): remove unwraps, adjust fmt
* fix(tql): address taplo issue
* feat(tql): update parse_tql_query logic
* fix(tql): change query parsing logic to use parser instead of delimiter
* fix(tql): add timestamp function support, add sqlness tests
* fix(tql): add lookback optional param for tql eval
* fix(tql): adjust tests for now() function
* fix(tql): introduce the tqlerror to differentiate failures on parsing, evaluation and simplification stages
* fix(tql): add tests for explain/analyze
* feat(tql): add lookback support for explain/analyze, update tests
* feat(tql): add more sqlness tests
* chore(tql): extract common logic for eval, analyze and explain into a single function
* feat(tql): address CR points
* feat(tql): use snafu for tql errors, add more docs
* feat(tql): address CR points
2024-03-29 06:37:25 +00:00
Eugene Tolbakov
8609977b52
feat: add verbose support for tql explain/analyze ( #3390 )
...
* feat: add verbose support for tql explain/analyze
* chore: apply clippy suggestions
* feat: add sqlness tests
* fix: adjust sqlness replace rules
* fix: address CR (move tql explain/analyze inside common folder)
* fix: address CR(improve comments to indicate that verbose is optional)
2024-03-02 11:18:22 +00:00
Ruihang Xia
jeremyhi
discord9
LFC
Ning SunandRuihang Xia
fys
dennis zhuang
discord9
Zhenchi
Yingwen
Eugene Tolbakov