Commit Graph
41 Commits
Author SHA1 Message Date
+1 c60311973d fix: backport v1.2.0 correctness and liveness fixes (#9023)
* fix(mito2): fence checkpoints during region transitions (#8847)

* fix: fence checkpoints during region transitions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(datanode): fix transient downgrade setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): fix checkpoint lifecycle test setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): cover cancelled downgrade waiter retry

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): fence direct follower transitions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: trim checkpoint transition coverage

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito2): clarify checkpoint task lifecycle

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit a502dfdefd)

* fix(mito2): split SSTs at primary key series boundaries (#8888)

* fix(mito2): split SSTs at series boundaries

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover SST splitting without primary key

Also document the sortedness precondition and the series boundary
split semantics on write_all_flat/write_all_flat_as_primary_key and
the new split helpers.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): avoid per-row Vec allocation for empty primary key

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Update src/mito2/src/sst/parquet/writer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
(cherry picked from commit 4ac3423261)

* fix(meta): allow manual migration from offline datanodes (#8934)

* fix(meta): allow migration from offline datanodes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: fix offline migration event actor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: read migration routes from metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 8a473c5bf0)

* fix(flight): bound DoGet response wait (#8943)

* fix(flight): defer datanode query initialization

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(client): retain Flight stream peer context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(client): improve Flight stream diagnostics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 28398138ec)

* fix(query): keep INSERT timestamp conversion out of the source query (#8911)

* fix(query): keep INSERT timestamp conversion out of the source query

Interpreting an INSERT's string timestamps used to work by pushing the
conversion down into the source query, which changed what that query
means. Two consequences:

- Pushing through a UNION's DISTINCT moved the dedup key from the raw
  strings to parsed instants, so rows spelling the same instant
  differently collapsed into one. On an append-only table that is a
  silently dropped row.
- A UNION branch that needed no conversion (a NULL, or an explicit cast)
  made the whole column give up, leaving sibling branches on UTC while
  the rest of the row used the session timezone.

Convert at the assignment instead, by routing its cast through a
timezone-carrying timestamp type and back. Arrow applies the timezone
when a cast target carries one, and stripping it afterwards preserves
the value. The source query is no longer touched, so both cases go away
and the tree-walking rewrite (roughly 160 lines) is deleted.

The rewrite reads source types, so it now runs TypeCoercion first: a
UNION still carries its loose per-branch schema before coercion, and
retargeting a cast whose input later becomes a timestamp would shift the
value rather than reinterpret it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(query): address review on INSERT assignment rewrite

- Clone the input `Arc` instead of the whole subtree, and only rebuild it
  when a `Values` row actually changes.
- Defer cloning the cast source until the literal-folding path has been
  ruled out.
- Move the UTC check onto `Timezone::is_utc`, replacing a bare string
  compare.
- Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types
  as `Null`, so the assignment cast is left for parameter substitution.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 1851f6bf4d)

* test: renew etcd TLS certificates (#8956)

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 144f83528d)

* fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)

* chore(deps): update tokio-postgres

* fix: describing fetch cursor and analyze

(cherry picked from commit b31f05eb59)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): avoid insert select HTTP/2 stalls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d4a3d88cd7)

* fix(query): share record batch forwarding for DML

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit c34bec197c)

* fix(mysql): interpret prepared statement datetime params in session timezone (#8923)

* fix(mysql): interpret prepared statement datetime params in session timezone

Binary DATETIME parameters of server-side prepared statements were
converted as if UTC, ignoring the session timezone set via SET time_zone.
Convert them with the session timezone and add an integration test
covering prepared inserts and predicates under Asia/Shanghai.

Signed-off-by: wy471x <wy471x@gmail.com>

* refactor: share naive datetime timezone policy via common-time

Address review feedback on the prepared-statement timezone fix:

- Expose Timestamp::from_naive_datetime in common-time so the DST policy
  (gap -> error, ambiguous -> earlier instant) lives in one place, shared
  by the text protocol (Timestamp::from_str) and the MySQL binary protocol.
- Route the MySQL prepared-statement datetime conversion through it.
- Match the target type before converting datetime params so
  PreparedStmtTypeMismatch fails fast without wasted conversion.
- Use the short Timezone import form for consistency with the rest of servers.

Signed-off-by: wy471x <wy471x@gmail.com>

---------

Signed-off-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
(cherry picked from commit aaa843104b)

* fix(mito2): use target sequence for foreign SSTs (#8946)

* fix(mito2): use target sequence for foreign SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): address foreign SST review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit bd7d2c1dfa)

* fix: postgres describe for more statements (#8974)

* fix: postgres describe for more statements

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: cover more show statements

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments

- add missing `clippy::too_many_arguments` allow on
  `query_from_information_schema_dataframe` (CI clippy failure)
- take `&ShowKind` in the information-schema dataframe helper so `kind`
  is no longer cloned at every call site; only the WHERE arm (which needs
  an owned expression for `sql_to_expr`) clones internally
- document why re-applying TQL explain formats never overwrites an
  existing value (per-query context state)

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: trim comments to essentials

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit d32cd77505)

* fix(promql): resolve derived labels in aggregation arithmetic (#8994)

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit c4dafb5826)

* fix(deps): narrow postgres lockfile update

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(deps): normalize remaining x509 lock reference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(client): isolate query and control transports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(frontend): clarify adaptive window purpose

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(grpc): satisfy clippy in transport isolation coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(pipeline): coalesce concurrent pipeline cache misses (#9022)

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "test(grpc): satisfy clippy in transport isolation coverage"

This reverts commit 888bcedc96.

(cherry picked from commit e0216b4074485bcffaae53348a17091532cb3600)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "docs(frontend): clarify adaptive window purpose"

This reverts commit 84cf69d099.

(cherry picked from commit da30e605ad7ebfb2a3fa86ea44b67405e5afde98)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "fix(client): isolate query and control transports"

This reverts commit a1d2dbb050.

(cherry picked from commit 03f5c31f544d3d3852beddfa87cff34d04a646ae)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(json2): add bounded auto-expansion to the JSON2 vector builder (#8909)

* refactor(json2): add bounded auto-expansion to the JSON2 vector builder

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 76f08d2b3f)
(cherry picked from commit 640fa63bd8ee098fac8e8d62d6fb7ca5af4ddcd0)

* refactor(json2): optimize JSON2 building without auto-expanded paths (#8928)

* refactor(json2): optimize JSON2 building without auto-expanded paths

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* avoid panicking memtable write

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 7fd0a7bb98)
(cherry picked from commit 63c68ee60101e2a09e14420051f8f889b41a2a3b)

* refactor(json2): support querying v2 storage layout (#8940)

* feat(json2): support querying v2 storage layout

- route missing JSON2 paths to the v2 remainder field
- reconstruct complete values from explicit fields and remainder data
- preserve root JSON2 columns across projections and filters
- support nested JSON values in json_get string results
- add and reorganize JSON2 sqlness coverage

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 932f87f7a8)
(cherry picked from commit 5af1394a1f319bb99cafa9474b8a5a075be1e805)

* feat(pipeline): support table-aware JSON2 transforms (#8964)

* feat(pipeline): support table-aware JSON2 transforms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(pipeline): support JSON2 type hints in transforms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(pipeline): default failed JSON2 transforms to null

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(json2): distinguish invalid settings from layout errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
(cherry picked from commit fb86f6573e)
(cherry picked from commit 18a21a9ce017b6887c857a3f3bbb9a83aa7f968b)

* refactor: json2 v2 storage layout (#8979)

* refactor: json2 v2 storage layout

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* rethinking when "needs_remainder"

Signed-off-by: luofucong <luofc@foxmail.com>

* restore "ReadColumns"

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 529f046110)
(cherry picked from commit 7628d795197c786872730c3a5ef36c5c662d4322)

* feat(json2): support JSON2 paths in SQL functions (#9007)

feat(query): support JSON2 paths in SQL functions

Update the DataFusion fork to expose scalar function planning hooks.

Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs.

Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions.

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 15317a131b)
(cherry picked from commit a945cbc7a4398e6bee74d4d2c67528772faf398d)

* feat(json2): support empty and null JSON2 value (#9010)

* feat(json2): support empty and null JSON2 value

Signed-off-by: fys <fengys1996@gmail.com>

* test(json2): cover explicit NULL and omitted-column inserts

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cargo fmt

Signed-off-by: fys <fengys1996@gmail.com>

* fix: infer empty JSON object as object type

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
(cherry picked from commit 05c65f54a8)
(cherry picked from commit 05c27452ea5e958fab85e0ba4b82f5df205a5d08)

* feat(json2): support list indexing for JSON2 columns (#9013)

feat(query): support list indexing for JSON2 columns

Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index.

Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation.

Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs.

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d62a5a990a)
(cherry picked from commit c11d8647feb30ecdd5ba1bc8ed4df78999801672)

* refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)

* refactor(json2): concretize JSON2 schemas at merge scan boundaries

Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec.

Add SQLness coverage for whole JSON2 columns in windows and joins.

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit 84bd993131)
(cherry picked from commit 51c98a7e31a929fc33ff5133714bb47108f7539b)

* fix(json2): keep empty structs in remainder (#9027)

Signed-off-by: luofucong <luofc@foxmail.com>
(cherry picked from commit d67d3501a9)
(cherry picked from commit b3f2c1cc5bf0cfddb7c0670ac8f3522eb67c35a0)

* fix(mito2): remove stale scan test imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit d63bbe6aef298f5b8e960bb02eed08743a6684ee)

* test(query): expect bounded merge scan partitions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 7bcef46b1c010d2e73ec49be7581c216e6b5b1b6)

* style(mito2): remove trailing whitespace

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(sqlness): refresh bounded partition plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(sqlness): normalize CPU-dependent partition plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: wy471x <wy471x@gmail.com>
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: wy471x <wy471x@gmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: fys <fengys1996@gmail.com>
2026-09-07 19:29:54 +08:00
discord9 5ed7128f06 perf(query): prune RangeSelect input projections (#8570)
* perf(query): prune RangeSelect input projections

Signed-off-by: discord9 <discord9@163.com>

* test(query): stabilize RangeSelect projection snapshot

Signed-off-by: discord9 <discord9@163.com>

* test(query): update RangeSelect nested plan snapshot

Signed-off-by: discord9 <discord9@163.com>

* fix(query): handle nested RangeSelect aliases

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-30 07:09:49 +00:00
Ruihang Xia 0d90f7407c fix: infer time index from column meta on derived table (#8013)
* rough fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reorganize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enhance default by infer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* supply comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-12 08:18:46 +00:00
b2074e3863 chore: upgrade DataFusion family, again (#7578)
* chore: upgrade DataFusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch to released version of datafusion-pg-catalog

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-03-03 07:36:39 +00:00
LFCandNing Sun 2f637a262e chore: update datafusion to 50 (#7076)
* chore: update datafusion to 50

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* fix: update datafusion_pg_catalog import

* chore: fix toml format

* chore: fix toml format again

* fix nextest

Signed-off-by: luofucong <luofc@foxmail.com>

* fix sqlness

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch datafusion-orc to upstream tag

* fix sqlness

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-10-23 07:18:36 +00:00
dennis zhuang 8e7f2e92cc test: adds approx_percentile_cont to range query test (#6903)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2025-09-04 12:18:27 +00:00
LFC f9d2a89a0c chore: update datafusion family (#6675)
* chore: update datafusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* use official otel-arrow-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* rebase

Signed-off-by: luofucong <luofc@foxmail.com>

* use the official orc-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* remove the empty lines

Signed-off-by: luofucong <luofc@foxmail.com>

* try following PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-08-15 12:41:49 +00:00
discord9 6501b0b13c feat: MergeScan print input (#6563)
* feat: MergeScan print input

Signed-off-by: discord9 <discord9@163.com>

* test: fix ut

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2025-07-21 06:40:15 +00:00
Ruihang Xia 7468a8ab2a feat: organize EXPLAIN ANALYZE VERBOSE's output in JSON format (#6308)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-06-12 09:55:53 +00:00
85d564b0fb fix: upgrade sqlparse and validate align in range query (#5958)
* fix: upgrade sqlparse and validate align in range query

* update sqlparser to the merged commit

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
2025-04-25 17:34:49 +00:00
LFC 311727939d chore: update datafusion family (#5814) 2025-04-09 02:20:55 +00:00
dennis zhuang d53afa849d fix: interval cast expression can't work in range query, #5805 (#5813)
* fix: interval cast expression can't work in range query, #5805

* fix: nested cast

* test: make vector test stable
2025-04-02 08:46:17 +00:00
Ruihang Xia e463942a5b fix: recover plan schema after dist analyzer (#5665)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-03-07 00:29:55 +00:00
35b635f639 feat!: Bump datafusion, prost, hyper, tonic, tower, axum (#5417)
* change dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: adapt to arrow's interval array

* chore: fix compile errors in datatypes crate

* chore: fix api crate compiler errors

* chore: fix compiler errors in common-grpc

* chore: fix common-datasource errors

* chore: fix deprecated code in common-datasource

* fix promql and physical plan related

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip: upgrading network deps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* block on updating `sqlparser`

* upgrade sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adapt new df's trait requirements

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix compiler errors in mito2

* chore: fix common-function crate errors

* chore: fix catalog errors

* change import path

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix some errors in query crate

* chore: fix some errors in query crate

* aggr expr and some other tiny fixes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix expr related errors in query crate

* chore: fix query serializer and admin command

* chore: fix grpc services

* feat: axum serve

* chore: fix http server

* remove handle_error handler
* refactor timeout layer
* serve axum

* chore: fix flow aggr functions

* chore: fix flow

* feat: fix errors in meta-srv

* boxed()
* use TokioIo

* feat!: Remove script crate and python feature (#5321)

* feat: exclude script crate

* chore: simplify feature

* feat: remove the script crate

* chore: remove python feature and some comments

* chore: fix warning

* chore: fix servers tests compiler errors

* feat: fix tests-integration errors

* chore: fix unused

* test: fix catalog test

* chore: fix compiler errors for crates using common-meta

testing feature is enabled when check with --workspace

* test: use display for logical plan test

* test: implement rewrite for ScanHintRule

* fix: http server build panic

* test: fix mito test

* fix: sql parser type alias error

* test: fix TestClient not listen

* test: some flow tests

* test(flow): more fix

* fix: test_otlp_logs

* test: fix promql test that using deprecated method fun()

* fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64

* test: fix infer schema test case

* test: fix tests related to plan display

* chore: fix last flow test

* test: fix function format related assertion

* test: use larger port range for tests

* fix: test_otlp_traces

* fix: test_otlp_metrics

* fix range query and dist plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: flow handle distinct use deprecated field

* fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs

* test: fix deserialize test

* test: reduce split key case num

* tests: lower case aggr func name

* test: fix some sqlness tests

* tests: more sqlness fix

* tests: fixed sqlness test

* commit non-bug changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: make our udf correct

* fix: implement empty methods of ContextProvider for DfContextProviderAdapter

* test: update sqlness test result

* chore: remove unused

* fix: provide alias name for AggregateExprBuilder in range plan

* test: update range query result

* fix: implement missing ContextProvider methods for DfContextProviderAdapter

* test: update timestamps, cte result

* fix: supports empty projection in mito

* test: update comment for cte test

* fix: support projection for numbers

* test: update test cases after projection fix

* fix: fix range select first_value/last_value

* fix: handle CAST and time index conflict

* fix: handle order by correctly in range first_value/last_value

* test: update sqlness result

* test: update view test result

* test: update decimal test

wait for https://github.com/apache/datafusion/pull/14126 to fix this

* feat: remove redundant physical optimization

todo(ruihang): Check if we can remove this.

* test: update sqlness test result

* chore: range select default sort use nulls_first = false

* test: update filter push down test result

* test: comment deciaml test to avoid different panic message

* test: update some distributed test result

* test: update test for distributed count and filter push down

* test: update subqueries test

* fix: SessionState may overwrite our UDFs

* chore: fix compiler errors after merging main

* fix: fix elasticsearch and dashboard router panic

* chore: fix common-functions tests

* chore: update sqlness result

* test: fix id keyword and update sqlness result

* test: fix flow_null test

* fix: enlarge thread size in debug mode to avoid overflow

* chore: fix warnings in common-function

* chore: fix warning in flow

* chore: fix warnings in query crate

* chore: remove unused warnings

* chore: fix deprecated warnings for parquet

* chore: fix deprecated warning in servers crate

* style: fix clippy

* test: enlarge mito cache tttl test ttl time

* chore: fix typo

* style: fmt toml

* refactor: reimplement PartialOrd for RangeSelect

* chore: remove script crate files introduced by merge

* fix: return error if sql option is not kv

* chore: do not use ..default::default()

* chore: per review

* chore: update error message in BuildAdminFunctionArgsSnafu

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* refactor: typed precision

* update sqlness view case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: flow per review

* chore: add example in comment

* chore: warn if parquet stats of timestamp is not INT64

* style: add a newline before derive to make the comment more clear

* test: update sqlness result

* fix: flow from substrait

* chore: change update_range_context log to debug level

* chore: move axum-extra axum-macros to workspace

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
2025-01-23 06:15:40 +00:00
discord9 4402f638cd fix: distinct respect in range (#5015)
* fix: distinct respect in range

* tests: sqlness

* chore: newline
2024-11-18 12:11:07 +00:00
2f2b4b306c feat!: implement interval type by multiple structs (#4772)
* define structs and methods

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: re-implement interval types in time crate

* feat: use new

* feat: interval value

* feat: query crate interval

* feat: pg and mysql interval

* chore: remove unused imports

* chore: remove commented codes

* feat: make flow compile but may not work

* feat: flow datetime

* test: fix some tests

* test: fix some flow tests(WIP)

* chore: some fix test&docs

* fix: change interval order

* chore: remove unused codes

* chore: fix cilppy

* chore: now signature change

* chore: remove todo

* feat: update error message

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: discord9 <discord9@163.com>
2024-10-14 03:09:03 +00:00
YingwenandRuihang Xia 3e17c09e45 feat: skip caching uncompressed pages if they are large (#4705)
* feat: cache each uncompressed page

* chore: remove unused function

* chore: log

* chore: log

* chore: row group pages cache kv

* feat: also support row group level cache

* chore: fix range count

* feat: don't cache compressed page for row group cache

* feat: use function to get part

* chore: log whether scan is from compaction

* chore: avoid get column

* feat: add timer metrics

* chore: Revert "feat: add timer metrics"

This reverts commit 4618f57fa2ba13b1e1a8dec83afd01c00ae4c867.

* feat: don't cache individual uncompressed page

* feat: append in row group level under append mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fetch pages cost

* perf: yield

* Update src/mito2/src/sst/parquet/row_group.rs

* refactor: cache key

* feat: print file num and row groups num in explain

* test: update sqlness test

* chore: Update src/mito2/src/sst/parquet/page_reader.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-09-10 11:52:16 +00:00
Yingwenanddennis zhuang f382a7695f perf: reduce lock scope and improve log (#4453)
* feat: refine logs for scan

* feat: improve build parts and  unordered scan metrics

* feat: change to debug log

* fix: release lock before reading part

* test: replace region id

* test: fix sqlness

* chore: add todo

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-31 04:07:34 +00:00
Ruihang Xia a7aa556763 feat: output multiple partition in MergeScanExec (#4227)
* feat: output multiple partition in MergeScanExec

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix range manipulate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-28 13:45:22 +00:00
LFC cc2f7efb98 chore: bump datafusion version to fix last_value regression (#4169)
* chore: bump datafusion version to fix `last_value` regression

* fix: resolve PR comments

* fix ci
2024-06-19 07:47:17 +00:00
Yingwen fe74efdafe feat: Implement memtable range (#4162)
* refactor: RangeBase

* feat: memtable range

* feat: scanner use mem range

* feat: remove base from mem range context

* feat: impl ranges for memtables

* chore: fix warnings

* refactor: make predicate cheap to clone

* refactor: MemRange -> MemtableRange

* feat: pub empty memtable to fix warnings

* test: fix sqlness result
2024-06-18 22:25:19 +00:00
YingwenandRuihang Xia 65f8b72d34 feat: Implement RegionScanner for SeqScan (#4060)
* feat: ordered builder wip

* feat: impl RegionScanner for SeqScan

* feat: implement scan_partition and build_stream

* chore: return SeqScan as RegionScanner

* fix: group parts

* feat: split parts

* chore: reader metrics

* chore: metrics

* chore: remove unused codes

* chore: support holding a group of ranges in ScanPart

* feat: group ScanParts to ScanParts

* feat: impl SeqScanner again

* chore: observe build cost in ScannerMetrics

* chore: fix compiler warnings

* style: fix clippy

* docs: update config docs

* chore: forward DisplayAs to scanner

* test: update sqlness tests

* chore: update debug fmt

* chore: custom debug for timestamp

fix test compiling issue with common-macro when running
cargo nextest -p common-time

* chore: update debug format

* feat: update fmt for scan part

* chore: fix warning

* fix: sanitize parallelism

* feat: split parts

* test: fix config api test

* feat: update logs

* chore: Revert "chore: remove unused codes"

This reverts commit b548b30a01eeded59b1a0a8d89f9293ca63afc41.

* chore: Revert "docs: update config docs"

This reverts commit a7997e78d6ddcf635560574de8c1948c495bdd12.

* feat: each partition scan files in parallel

* test: fix config api test

* docs: fix typo

* chore: address comments, simplify tests

* feat: global semaphore

* feat: always spawn task

* chore: simplify default explain output format

* handle output partiton number is 0

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-12 08:21:30 +00:00
Ruihang XiaandJeremyhi c2218f8be8 build(deps): bump datafusion 20240528 (#4061)
* build(deps): bump datafusion 20240528

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* another update

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update expected sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix first/last value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reformat comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remaining errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert toml format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 feature

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dead code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* format file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
2024-06-01 14:03:00 +00:00
Yingwen 179c8c716c feat: Adds RegionScanner trait (#3948)
* feat: define region scanner

* feat: single partition scanner

* feat: use single partition scanner

* feat: implement ExecutionPlan wip

* feat: mito engine returns single partition scanner

* feat: implement DisplayAs for region server

* feat: dummy table provider use handle_partitioned_query()

* test: update sqlness test

* feat: table provider use ReadFromRegion

* refactor: remove StreamScanAdapter

* chore: update lock

* style: fix clippy

* refactor: remove handle_query from the RegionEngine trait

* chore: address CR comments

* refactor: rename methods

* refactor: rename ReadFromRegion to RegionScanExec
2024-05-20 11:52:00 +00:00
WU Jingdi e2a1cb5840 feat: support evaluate expr in range query param (#3823)
* feat: support evaluate expr in range query param

* chore: fix comment

* chore: fix code comment

* fix: disbale now in duration param
2024-05-17 08:31:55 +00:00
Ruihang XiaandJeremyhi aec5cca2c7 feat: support distributed EXPLAIN ANALYZE (#3908)
* feat: fetch and pass per-plan metrics

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl DistAnalyzeExec

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/query/src/analyze.rs

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
2024-05-10 12:31:29 +00:00
Ruihang Xia e84b1eefdf perf: optimize count(*) (#3845)
* perf: optimize count(*)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fallback to count(1) for temporary table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias expr in range plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle subquery alias

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-30 10:10:11 +00:00
314f2704d4 build(deps): update datafusion to latest and arrow to 51.0 (#3661)
* chore: update datafusion

* update sqlness case of time.sql

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: adjust range query partition

* fix: hisogram incorrect result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: ignore filter pushdown temporarily

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update limit sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: histogram with wrong distribution

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update negative ordinal sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: bump df to cd7a00b

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* ignore test_range_filter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix promql exec panic

* fix "select count(*)" exec error

* re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed

* fix: range query schema error

* update sqlness results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* update datafusion, again

* fix pyo3 compile error, and update some sqlness results

* update decimal sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: promql literal

* fix udaf tests

* fix filter pushdown sqlness tests

* fix?: test_cast

* fix: rspy test fail due to datafusion `sin` signature change

* rebase main to see if there are any failed tests

* debug ci

* debug ci

* debug ci

* enforce input partition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* debug ci

* fix ci

* fix ci

* debug ci

* debug ci

* debug ci

* fix sqlness

* feat: do not return error while creating a filter

* chore: remove array from error

* chore: replace todo with unimplemented

* Update src/flow/clippy.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WUJingdi <taylor-lagrange@qq.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-04-18 12:07:18 +00:00
WU Jingdi fb4da05f25 fix: adjust fill behavior of range query (#3489) 2024-03-13 09:20:34 +00:00
WU Jingdi 8c37c3fc0f feat: support first_value/last_value in range query (#3448)
* feat: support `first_value/last_value` in range query

* chore: add sqlness test on `count`

* chore: add test
2024-03-11 01:30:39 +00:00
dennis zhuang 7323e9b36f feat: change Range Query’s default align behavior aware of timezone (#3219)
* feat: change Range Query’s default align behavior to 1970-01-01 00:00:00 aware of timezone

* test: test with +23:00 timezone
2024-01-24 08:17:57 +00:00
WU Jingdi d180e41230 fix: change range query time slot to [align_ts, align_ts + range) (#2938) 2023-12-19 02:35:14 +00:00
WU Jingdi 4383a69876 fix!: remove range calendar as To option (#2940)
fix: remove range calendar as `To` option
2023-12-18 07:48:49 +00:00
WU Jingdi 806400caff feat: add align to / interval support in range query (#2842)
* feat: add align to / interval support in range query

* chore: fix ci

* chore: simplify `parse_duration_expr`

* chore: change s to ms
2023-12-04 08:00:41 +00:00
Ruihang Xia 56fc77e573 fix: add missing error display message (#2791)
* fix: add missing error display message

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-11-23 02:59:49 +00:00
WU JingdiandRuihang Xia 91820a8006 fix: empty by in range query (#2770)
* fix: empty by in range query

* Apply suggestions from code review

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2023-11-17 11:18:14 +00:00
WU Jingdi bbaae9223a feat: enable empty range by (#2697) 2023-11-06 03:45:33 +00:00
WU Jingdi d2f3793d15 fix: explain range query (#2664)
* fix: explain range query

* chore: add comment
2023-10-30 13:21:25 +00:00
Ruihang Xia 5d8b0e8154 chore: bump dependencies (#2659)
* fix all hard error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* trivial changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix order by

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sql keyword and data type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix range exec's input partitioning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix cover input type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix explain analyze

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update distributed mode sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update locks

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* downgrade dlv-list

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-10-26 12:24:14 +00:00
Ning Sun 0fbde48655 feat: hide internal error and unknown error message from end user (#2544)
* feat: use fixed error message for unknown error

* feat: return fixed message for internal error as well

* chore: include status code in error message

* test: update tests for asserts of error message

* feat: change status code of some datafusion error

* fix: make CollectRecordbatch an query error

* test: update sqlness results
2023-10-23 03:07:35 +00:00
WU Jingdi 46e106bcc3 feat: allow nest range expr in Range Query (#2557)
* feat: eable range expr nest

* fix: change range expr rewrite format

* chore: organize range query tests

* chore: change range expr name(e.g. MAX(v) RANGE 5s FILL 6)

* chore: add range query test

* chore: fix code advice

* chore: fix ca
2023-10-18 07:03:26 +00:00