Commit Graph

384 Commits

Author SHA1 Message Date
Ruihang Xia
2d992f4f12 fix: check_partition uses unqualified name (#4452)
* fix: check_partition uses unqualified name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-07-30 11:28:28 +00:00
taobo
1138f32af9 feat: support setting time range in Copy From statement (#4405)
* feat: support setting time range in Copy From statement

* test: add batch_filter_test

* fix: ts data type inconsistent error

* test: add sqlness test for copy from with statement

* fix: sqlness result error

* fix: cr comments
2024-07-29 16:55:19 +00:00
JohnsonLee
9a5fa49955 feat: support pg_namespace, pg_class and related psql command (#4428)
* feat: add function 'pg_catalog.pg_table_is_visible'q

* feat: add 'pg_class' and 'pg_namespace', now we can run '\d' and '\dt'!

* refactor: move memory_table::tables to utils::tables

* refactor: move out predicate to system_schema to reuse it

* feat: predicates pushdown

* test: add pg_namespace, pg_class related sqlness test

* fix: typos and license header

* fix: sqlness test

* refactor: use `expect` instead of `unwrap` here

* refactor: remove the `information_schema::utils` mod

* doc: make the comment in pg_get_userbyid more precise

* doc: add TODO and comment in pg_catalog

* fix: typo

* fix: sqlness

* doc: change to comment on PGClassBuilder to TODO
2024-07-28 12:04:54 +00:00
discord9
021ec7b6ac feat(flow): flush_flow function (#4416)
* refactor: df err variant

* WIP

* chore: update proto version

* chore: revert mistaken rust-toolchain

* feat(WIP): added FlowService to QueryEngine

* refactor: move flow service to operator

* refactor: flush use flow name not id

* refactor: use full path in macro

* feat: flush flow

* feat: impl flush flow

* chore: remove unused

* chore: meaninful response

* chore: remove unused

* chore: clippy

* fix: flush_flow with proper blocking

* test: sqlness tests added back for flow

* test: better predicate for flush_flow

* refactor: rwlock

* fix: flush lock

* fix: flush lock write then drop

* test: add a new flow sqlness test

* fix: sqlness testcase

* chore: style

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-26 23:04:13 +00:00
dennis zhuang
0710e6ff36 fix: remove to_timezone function (#4439)
fix: remove to_timezone, it doesn't make sense
2024-07-26 07:40:07 +00:00
dennis zhuang
db3a07804e fix: information_schema tables and views column value (#4438) 2024-07-26 07:39:58 +00:00
discord9
9fa9156bde feat: FLOWS table in information_schema&SHOW FLOWS (#4386)
* feat(WIP): flow info table

refactor: better err handling&log

feat: add flow metadata to info schema provider

feat(WIP): info_schema.flows

feat: info_schema.flows table

* fix: err after rebase

* fix: wrong comparsion op

* feat: SHOW FLOWS&tests

* refactor: per review

* chore: unused

* refactor: json error

* chore: per review

* test: sqlness

* chore: rm inline error

* refactor: per review
2024-07-19 09:29:36 +00:00
discord9
c595a56ac8 test(flow): ignore flow tests for now (#4377)
* tests: ignore flow tests for now

* chore: typo

* test: remove file altogether

* fix: last_value sqlness test regex

* fix: last_value part REDACTED
2024-07-16 10:07:21 +00:00
dennis zhuang
be3ea0fae7 feat: improve datafusion external error and mysql error (#4362)
* feat: improve datafusion external error and mysql error

* chore: address CR comments and fix tests

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
2024-07-16 07:01:09 +00:00
discord9
b2c5f8eefa test: more sleep when flow insert makes it serial (#4373)
* tests: more sleep

* fix: wait more time

* fix: update result file
2024-07-16 03:36:31 +00:00
JohnsonLee
072d7c2022 feat: introduce 'pg_catalog.pg_type' (#4332)
* WIP: pg_catalog

* refactor: move memory_table to crate public level to reuse it in pgcatalog

* refactor: new system_schema mod to manage implementation of information_schema and pg_catalog

* feat: pg_catalog.pg_type

* fix: remove unused code to avoid warning

* test: add pg_catalog sqlness test

* feat: pg_catalog_cache in system_catalog

* fix: integration test

* test: rollback unit test

* refactor: mix pg_catalog table_id with old ones

* fix: add todo information

* tests: rerun sqlness

---------

Co-authored-by: johnsonlee <johnsonlee@localhost.localdomain>
2024-07-15 17:41:08 +00:00
Yingwen
7900367433 test: replace unstable output of last value test (#4371)
test: replace unstable output
2024-07-15 16:51:16 +00:00
discord9
2b912d93fb feat: flow perf&fix df func call (#4347)
* feat: flow perf&fix df func call

feat: make source sender `send` non-blocking

feat: better control of flow worker freq

feat: support transform interval

fix: const folding df func args&tests

tests: update cast const fold

chore: adjust flow work's freq

refactor: batch split

feat: adaptive run freq flow worker&check for errors

chore: better debug log

* refactor: per review

* chore: per zc's review

* chore: per bot review

* chore: remove some `TODO` completed

* docs: add comments for a test
2024-07-15 09:20:04 +00:00
Yingwen
20d9c0a345 fix: scan hint checks order asc (#4365)
* fix: order by asc check

* feat: print selector in explain

* test: move last_value opt test to standalone

* test: sqlness remove space

* test: update regex for datetime

* test: fix partitioning

* chore: update comment

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-07-15 07:49:14 +00:00
Lanqing Yang
9501318ce5 feat: support show views statement (#4360)
This commit enables show view statement which will display a list of views
names.
2024-07-15 07:24:27 +00:00
Lanqing Yang
15ac8116ea feat: adding information_schema.views table (#4342)
This commit introduces information_schema.views table. The VIEWS table provides
information about views in databases.
2024-07-14 09:50:19 +00:00
Ruihang Xia
16075ada67 feat: impl optimizer rule to handle last_value case (#4357)
* feat: impl optimizer rule to handle last_value case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/query/src/optimizer/scan_hint.rs

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* split methods

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
2024-07-12 14:36:23 +00:00
tison
285ffc5850 fix: build info should use build time env var (#4343)
* fix: build info should use build time env var

Signed-off-by: tison <wander4096@gmail.com>

* catch up

Signed-off-by: tison <wander4096@gmail.com>

* fixup lifetime

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fix more

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2024-07-12 03:08:30 +00:00
dennis zhuang
ab22bbac84 feat: impl drop view (#4231)
* feat: impl drop view

* fix: metric name

* fix: comments

* test: add DropViewProcedure test

* test: drop view meets a table

* test: update sqlness tests by drop view

* feat: apply suggestion from AI

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* fix: TYPE_NAME for DropFlowProcedure

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
2024-07-11 19:53:54 +00:00
Ruihang Xia
374cfe74bf fix(sqlness): relax start time regex to match various precisions (#4326)
* fix(sqlness): relax start time regex to match various precisions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* exclude ip pattern

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-07-10 06:35:55 +00:00
dennis zhuang
33ed745049 feat: show create view and creating view with columns (#4086)
* feat: parse column names when creating view

* feat: save the view definition into view info

* feat: supports view columns and show create view

* feat: save plan columns for validation

* fix: typo

* chore: comments and style

* chore: apply suggestions

* test: assert CreateView display result

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: avoid the clone

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: compile error after rebeasing

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
2024-07-09 18:35:11 +00:00
discord9
1ddf19d886 feat: flownode use Inserter to write to database (#4323)
* feat: use `Inserter` as Frontend

* fix: enable procedure in flownode

* docs: remove `frontend_addr` opts

* chore: rm fe addr in test runner

* refactor: int test also use inserter invoker

* feat: flow shutdown&refactor: remove `Frontendinvoker`

* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`

* refactor: per review

* refactor: remove a layer of  box

* fix: standalone use `node_manager`

* fix: remove a `Arc` cycle
2024-07-09 10:44:22 +00:00
Ruihang Xia
185953e586 fix: support unary operator in default value, partition rule and prepare statement (#4301)
* handle unary operator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add prepare test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test and context

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rebase error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-09 08:59:06 +00:00
Ruihang Xia
23bb9d92cb feat: handle parentheses with unary ops (#4290)
* feat: handle parentheses with unary ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check tokens before convert to RPN

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test cases to sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-09 04:08:36 +00:00
dennis zhuang
f1d17a8ba5 fix: panic while reading information_schema. KEY_COLUMN_USAGE (#4318)
* fix: table might be dropped during iteration

* fix: panic while reading information_schema.key_column_usage

* fix: key_column_usage wrong results
2024-07-09 03:30:14 +00:00
Zhenchi
00308218b3 feat(fulltext_index): allow enable full-text index in SQL and gRPC way (#4310)
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: test_fulltext_intm_path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: explicitly build column options

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: fix error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
2024-07-08 20:18:48 +00:00
Ruihang Xia
96861137b2 fix(ci): remove sqlness state in success (#4313)
* fix(ci): remove sqlness state in success

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix regex

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-08 12:32:36 +00:00
dennis zhuang
bb32230f00 feat: impl show table status (#4303)
* feat: impl show table status

* chore: style and comment

* test: revert lz4 compression
2024-07-08 03:58:29 +00:00
Eugene Tolbakov
3f4928effc feat(sql): add iso-8601 format support for intervals (#4291)
* feat(sql): add iso-8601 format support for intervals

* fix(sql): address CR feedback

* chore(sql): use regex to check the start of iso 8601
2024-07-05 22:19:05 +00:00
discord9
09fff24ac4 feat: make flow distributed work&tests (#4256)
feat: flownode frontend client&test

feat: Frontend Client

feat: set frontend invoker for flownode

feat: set frontend invoker for flownode

chore: test script

WIP: test flow distributed

feat: hard coded demo

docs: flownode example toml

feat: add flownode support in runner

docs: comments for node

chore: after rebase

docs: add a todo

tests: move flow tests to common

fix: flownode sqlness dist test

chore: per review

docs: make

fix: make doc
2024-07-05 14:46:44 +00:00
Yingwen
60f599c3ef feat: expose merge_mode option (#4289)
feat: expose merge mode options
2024-07-05 07:40:01 +00:00
discord9
c21e969329 fix: call df_func with literal (#4265)
* fix: call df_func with literal

* chore: rm dbg log forget to remove
2024-07-05 06:21:22 +00:00
discord9
967b2cada6 feat!: remove alias metasrv-addr (#4239) 2024-07-03 06:53:43 +00:00
Yohan Wal
c4db9e8aa7 fix!: forbid to change information_schema (#4233)
* fix: forbid to change tables in information_schema

* refactor: use unified read-only check function

* test: add more sqlness tests for information_schema

* refactor: move is_readonly_schema to common_catalog
2024-07-03 03:09:23 +00:00
Ning Sun
11cf9c827e feat: dbeaver mysql compatibility, use statement and information_schema.tables (#4218)
* feat: add more placeholder field in information_schema.tables

* feat: make schema modifiable for use statement

* chore: add todo items

* fix: resolve lint issues after data type changes

* chore: update sqlness results

* refactor: patch for select database is no longer needed

* test: align tests and data types

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* fix: use canonicalize_identifier for database name

* feat: add all columns for information_schema.tables

* test: remove vairables from sqlness results

* feat: add to_string impl for table options

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-03 01:31:13 +00:00
Ruihang Xia
a7aa556763 feat: output multiple partition in MergeScanExec (#4227)
* feat: output multiple partition in MergeScanExec

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix range manipulate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-28 13:45:22 +00:00
Lei, HUANG
ef935a1de6 feat!: reduce sorted runs during compaction (#3702)
* feat: add functions to find and merge sorted runs

* chore: refactor code

* chore: remove some duplicates

* chore: remove one clone

* refactor: change max_active_window_files to max_active_window_runs

* feat: integrate with sorted runs

* fix: unit tests

* feat: limit num of sorted runs during compaction

* fix: some test

* fix: some cr comments

* feat: use smallvec

* chore: rebase main

* feat/reduce-sorted-runs:
 Refactor compaction logic and update test configurations

 - Refactored `merge_all_runs` function to use `sort_ranged_items` for sorting.
 - Improved item merging logic by iterating with `into_iter` and handling overlaps.
 - Updated test configurations to use `max_active_window_runs` instead of `max_active_window_files` for consistency.

---------

Co-authored-by: tison <wander4096@gmail.com>
2024-06-28 08:17:30 +00:00
Eugene Tolbakov
8702066967 feat(sql): add casting support for shortened intervals (#4220)
* feat(sql): add casting support for shortened intervals

* chore(sql): apply CR suggestion, minor renamings
2024-06-26 22:07:09 +00:00
Ruihang Xia
4d4a6cd265 feat: validate partition rule on create table (#4213)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-25 12:55:01 +00:00
Eugene Tolbakov
840f81e0fd fix(sql): improve compound signed number processing (#4200) 2024-06-25 04:01:46 +00:00
Eugene Tolbakov
cdd4baf183 feat(sql): improve interval expression, support shortened version (#4182)
* feat(sql): improve interval expression, support shortened version

* fix(sql): remove accidental change of sqlness assertion

* fix(sql): address CR feedback, add more tests

* chore(sql): add more tests
2024-06-24 20:29:34 +00:00
taobo
77904adaaf fix: region_peers returns same region_id for multi logical tables (#4190)
* fix: `region_peers` returns same region_id for multi logical tables

* test: add sqlness test for information_schema.region_peers

* refactor: region_peers sqlness
2024-06-24 14:12:36 +00:00
discord9
517917453d fix(flow): fix call df func bug&sqlness test (#4165)
* tests: flow sqlness tests

* tests: WIP df func test

* fix: use schema before expand for transform expr

* tests: some basic flow tests

* tests: unit test

* chore: dep use rev not patch

* fix: wired sqlness error?

* refactor: per review

* fix: temp sqlness bug

* fix: use fixed sqlness

* fix: impl drop as async shutdown

* refactor: per bot's review

* tests: drop worker handler both sync/async

* docs: add rationale for test

* refactor: per review

* chore: fmt
2024-06-24 07:52:45 +00:00
Ruihang Xia
fce65c97e3 feat: make RegionScanner aware of PartitionRange (#4170)
* define PartitionRange

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add optimizer rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl aggr stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add fallback method

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document and rename struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-21 09:54:22 +00:00
dennis zhuang
4306cba866 feat: show database options (#4174)
* test: test create table with database ttl

* feat: show database options

* fix: comment

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: fix CR comments and refactor

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2024-06-20 04:21:58 +00:00
LFC
cc2f7efb98 chore: bump datafusion version to fix last_value regression (#4169)
* chore: bump datafusion version to fix `last_value` regression

* fix: resolve PR comments

* fix ci
2024-06-19 07:47:17 +00:00
Yingwen
fe74efdafe feat: Implement memtable range (#4162)
* refactor: RangeBase

* feat: memtable range

* feat: scanner use mem range

* feat: remove base from mem range context

* feat: impl ranges for memtables

* chore: fix warnings

* refactor: make predicate cheap to clone

* refactor: MemRange -> MemtableRange

* feat: pub empty memtable to fix warnings

* test: fix sqlness result
2024-06-18 22:25:19 +00:00
Weny Xu
d6392acd65 fix(sqlness): catch different format timestamp (#4149) 2024-06-15 11:13:01 +00:00
Yingwen
65f8b72d34 feat: Implement RegionScanner for SeqScan (#4060)
* feat: ordered builder wip

* feat: impl RegionScanner for SeqScan

* feat: implement scan_partition and build_stream

* chore: return SeqScan as RegionScanner

* fix: group parts

* feat: split parts

* chore: reader metrics

* chore: metrics

* chore: remove unused codes

* chore: support holding a group of ranges in ScanPart

* feat: group ScanParts to ScanParts

* feat: impl SeqScanner again

* chore: observe build cost in ScannerMetrics

* chore: fix compiler warnings

* style: fix clippy

* docs: update config docs

* chore: forward DisplayAs to scanner

* test: update sqlness tests

* chore: update debug fmt

* chore: custom debug for timestamp

fix test compiling issue with common-macro when running
cargo nextest -p common-time

* chore: update debug format

* feat: update fmt for scan part

* chore: fix warning

* fix: sanitize parallelism

* feat: split parts

* test: fix config api test

* feat: update logs

* chore: Revert "chore: remove unused codes"

This reverts commit b548b30a01eeded59b1a0a8d89f9293ca63afc41.

* chore: Revert "docs: update config docs"

This reverts commit a7997e78d6ddcf635560574de8c1948c495bdd12.

* feat: each partition scan files in parallel

* test: fix config api test

* docs: fix typo

* chore: address comments, simplify tests

* feat: global semaphore

* feat: always spawn task

* chore: simplify default explain output format

* handle output partiton number is 0

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-12 08:21:30 +00:00
LFC
5a6021e34f refactor: remove substrait ser/de for region query in standalone (#3812)
* refactor: remove substrait serde for region query in standalone

* fix ci

* move QueryRequest to common-query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: format toml

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-06-11 09:33:47 +00:00