rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-19 06:20:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,993 Commits 133 Branches 253 Tags
cb94bd45d3b8f18203a930badce7bb4b10da0c48
Commit Graph

845 Commits

Author SHA1 Message Date
Ning Sun
b298b35b3b feat: show root cause and db name on the error line (#4442) 
* feat: show root cause on the error line

* feat: show root error for grpc

* feat: add error information for http error

* feat: add db information on error mysql/postgres logs
 2024-07-29 03:59:42 +00:00
JohnsonLee
9a5fa49955 feat: support pg_namespace, pg_class and related psql command (#4428) 
* feat: add function 'pg_catalog.pg_table_is_visible'q

* feat: add 'pg_class' and 'pg_namespace', now we can run '\d' and '\dt'!

* refactor: move memory_table::tables to utils::tables

* refactor: move out predicate to system_schema to reuse it

* feat: predicates pushdown

* test: add pg_namespace, pg_class related sqlness test

* fix: typos and license header

* fix: sqlness test

* refactor: use `expect` instead of `unwrap` here

* refactor: remove the `information_schema::utils` mod

* doc: make the comment in pg_get_userbyid more precise

* doc: add TODO and comment in pg_catalog

* fix: typo

* fix: sqlness

* doc: change to comment on PGClassBuilder to TODO
 2024-07-28 12:04:54 +00:00
discord9
021ec7b6ac feat(flow): flush_flow function (#4416) 
* refactor: df err variant

* WIP

* chore: update proto version

* chore: revert mistaken rust-toolchain

* feat(WIP): added FlowService to QueryEngine

* refactor: move flow service to operator

* refactor: flush use flow name not id

* refactor: use full path in macro

* feat: flush flow

* feat: impl flush flow

* chore: remove unused

* chore: meaninful response

* chore: remove unused

* chore: clippy

* fix: flush_flow with proper blocking

* test: sqlness tests added back for flow

* test: better predicate for flush_flow

* refactor: rwlock

* fix: flush lock

* fix: flush lock write then drop

* test: add a new flow sqlness test

* fix: sqlness testcase

* chore: style

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-26 23:04:13 +00:00
dennis zhuang
0710e6ff36 fix: remove to_timezone function (#4439) 
fix: remove to_timezone, it doesn't make sense
 2024-07-26 07:40:07 +00:00
Ran Miller
918be099cd docs(common_error): format enum StatusCode docs (#4427) 
* fix: format comments end with . symbol
* docs: add commnet for RegionReadonly
* fix: comment error for DatabaseAlreadyExists
 2024-07-24 15:54:35 +00:00
zyy17
2ae2a6674e refactor: add get_storage_path() and get_catalog_and_schema() (#4397) 
refactor: add get_storage_path() and get_catalog_and_schema()
 2024-07-20 01:55:48 +00:00
discord9
9fa9156bde feat: FLOWS table in information_schema&SHOW FLOWS (#4386) 
* feat(WIP): flow info table

refactor: better err handling&log

feat: add flow metadata to info schema provider

feat(WIP): info_schema.flows

feat: info_schema.flows table

* fix: err after rebase

* fix: wrong comparsion op

* feat: SHOW FLOWS&tests

* refactor: per review

* chore: unused

* refactor: json error

* chore: per review

* test: sqlness

* chore: rm inline error

* refactor: per review
 2024-07-19 09:29:36 +00:00
Weny Xu
e39f49fe56 fix: ensure keep alive is completed in time (#4349) 
* fix: ensure keep alive is completed in time

* chore: apply suggestions from CR

* chore: use write runtime

* refactor: set META_LEASE_SECS to 5

* chore: set etcd replicas to 1

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* fix: set `MissedTickBehavior::Delay`

* chore: apply suggestions from CR
 2024-07-17 06:14:45 +00:00
dennis zhuang
be3ea0fae7 feat: improve datafusion external error and mysql error (#4362) 
* feat: improve datafusion external error and mysql error

* chore: address CR comments and fix tests

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
 2024-07-16 07:01:09 +00:00
JohnsonLee
072d7c2022 feat: introduce 'pg_catalog.pg_type' (#4332) 
* WIP: pg_catalog

* refactor: move memory_table to crate public level to reuse it in pgcatalog

* refactor: new system_schema mod to manage implementation of information_schema and pg_catalog

* feat: pg_catalog.pg_type

* fix: remove unused code to avoid warning

* test: add pg_catalog sqlness test

* feat: pg_catalog_cache in system_catalog

* fix: integration test

* test: rollback unit test

* refactor: mix pg_catalog table_id with old ones

* fix: add todo information

* tests: rerun sqlness

---------

Co-authored-by: johnsonlee <johnsonlee@localhost.localdomain>
 2024-07-15 17:41:08 +00:00
dennis zhuang
64cad4e891 feat: tweak error and status codes (#4359) 
* feat: tweak status codes

* fix: typo

* fix: by cr comments
 2024-07-15 07:50:16 +00:00
Ning Sun
b8bd8456f0 fix: remove path label for cache store (#4336) 
* fix: remove path label for cache store

* fix: ignore path label for intermediatemanager

* refactor: remove unused object store
 2024-07-15 03:34:19 +00:00
Lanqing Yang
15ac8116ea feat: adding information_schema.views table (#4342) 
This commit introduces information_schema.views table. The VIEWS table provides
information about views in databases.
 2024-07-14 09:50:19 +00:00
irenjj
9f2d53c3df refactor: Remove the StandaloneKafkaConfig struct (#4253) 
* refactor: Remove the StandaloneKafkaConfig struct

* remove the redundant assignment

* remove rudundant struct

* simplify replication_factor

* add KafkaTopicConfig

* fix check

* fix check

* fix check

* add flatten with

* revert config.md

* fix test params

* fix test param

* fix missing params when provider is kafka

* remove unsed files

* remove with prefix

* fix doc

* fix test

* fix clippy
 2024-07-12 08:17:18 +00:00
tison
285ffc5850 fix: build info should use build time env var (#4343) 
* fix: build info should use build time env var

Signed-off-by: tison <wander4096@gmail.com>

* catch up

Signed-off-by: tison <wander4096@gmail.com>

* fixup lifetime

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fix more

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-07-12 03:08:30 +00:00
dennis zhuang
ab22bbac84 feat: impl drop view (#4231) 
* feat: impl drop view

* fix: metric name

* fix: comments

* test: add DropViewProcedure test

* test: drop view meets a table

* test: update sqlness tests by drop view

* feat: apply suggestion from AI

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* fix: TYPE_NAME for DropFlowProcedure

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-07-11 19:53:54 +00:00
Lei, HUANG
da0c840261 feat: customize copy to parquet parameter (#4328) 
* feat/copy-to-parquet-parameter: Commit Message:

Enhance Parquet Writer with Column-wise Configuration

Summary:

 • Introduced column_wise_config function to customize per-column properties in Parquet writer.

* feat/copy-to-parquet-parameter: Commit Message:

Enhance Parquet File Format Handling for Specific Data Types

Summary:

 • Added ConcreteDataType import to support specific data type handling.

* feat/copy-to-parquet-parameter: Commit Message:

 Refactor Parquet file format configuration

* feat/copy-to-parquet-parameter:
 Enhance Parquet file format handling for timestamp columns

 - Added logic to disable dictionary encoding and set DELTA_BINARY_PACKED encoding for timestamp columns in the Parquet file format configuration.

* feat/copy-to-parquet-parameter:
 Disable dictionary encoding for timestamp columns in Parquet writer and update default max_active_window_runs in TwcsOptions

 - Modified Parquet writer to disable dictionary encoding for timestamp columns to optimize for increasing timestamp data.

* feat/copy-to-parquet-parameter:
 Update compaction settings in tests

 - Modified `test_compaction_region` to include new compaction options: `compaction.type`,
 `compaction.twcs.max_active_window_runs`, and `compaction.twcs.max_inactive_window_runs`.
 - Updated `test_merge_mode_compaction` to use `compaction.twcs.max_active_window_runs` and
 `compaction.twcs.max_inactive_window_runs` instead of `max_active_window_files` and
 `max_inactive_window_files`.
 2024-07-10 07:52:30 +00:00
Weny Xu
52a9a748a1 revert: lz4 compression (#4329) 
* Revert "test: revert lz4 compression"

This reverts commit 180dda13fa.

* refactor: remove compression field
 2024-07-10 04:24:40 +00:00
dennis zhuang
33ed745049 feat: show create view and creating view with columns (#4086) 
* feat: parse column names when creating view

* feat: save the view definition into view info

* feat: supports view columns and show create view

* feat: save plan columns for validation

* fix: typo

* chore: comments and style

* chore: apply suggestions

* test: assert CreateView display result

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: avoid the clone

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: compile error after rebeasing

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-09 18:35:11 +00:00
discord9
1ddf19d886 feat: flownode use Inserter to write to database (#4323) 
* feat: use `Inserter` as Frontend

* fix: enable procedure in flownode

* docs: remove `frontend_addr` opts

* chore: rm fe addr in test runner

* refactor: int test also use inserter invoker

* feat: flow shutdown&refactor: remove `Frontendinvoker`

* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`

* refactor: per review

* refactor: remove a layer of  box

* fix: standalone use `node_manager`

* fix: remove a `Arc` cycle
 2024-07-09 10:44:22 +00:00
Ruihang Xia
185953e586 fix: support unary operator in default value, partition rule and prepare statement (#4301) 
* handle unary operator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add prepare test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test and context

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rebase error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-09 08:59:06 +00:00
Ning Sun
7fe3f496ac refactor: do not print error log on PlanQuery error (#4322) 2024-07-09 06:34:30 +00:00
Weny Xu
1a9314a581 feat: enhanced the retry logic by adding a random noise (#4320) 
feat: enhanced the retry logic by adding a random noise to the retry delay to avoid retry storms
 2024-07-09 04:30:10 +00:00
Ruihang Xia
23bb9d92cb feat: handle parentheses with unary ops (#4290) 
* feat: handle parentheses with unary ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check tokens before convert to RPN

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test cases to sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-09 04:08:36 +00:00
Zhenchi
00308218b3 feat(fulltext_index): allow enable full-text index in SQL and gRPC way (#4310) 
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: test_fulltext_intm_path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: explicitly build column options

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: fix error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-08 20:18:48 +00:00
dennis zhuang
bb32230f00 feat: impl show table status (#4303) 
* feat: impl show table status

* chore: style and comment

* test: revert lz4 compression
 2024-07-08 03:58:29 +00:00
Weny Xu
bc398cf197 feat(remote wal): set default compresion to LZ4 (#4294) 
* feat(remote wal): set default compresion to LZ4

* fix: fix test
 2024-07-05 20:40:18 +00:00
Ning Sun
8399dcada3 refactor: use rwlock for modifiable session data (#4232) 
* chore: update sqlness results

* refactor: use rwlock for modifiable data in session and querycontext

* chore: format toml

* refactor: use mutable_inner structure for mutable fields

* refactor: remove arc wrapper
 2024-07-04 12:53:25 +00:00
Weny Xu
4ec247f34d feat: store peer info in TableFlowValue (#4280) 
* feat: store peer info in `TableFlowValue`

* chore: apply suggestions from CR
 2024-07-04 09:37:23 +00:00
Ruihang Xia
03c933c006 feat: handle AND/OR and priority in matches fn (#4270) 
* feat: handle AND/OR and priority in matches fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* transform AST

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle non-big-write AND & OR

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-04 06:19:03 +00:00
Weny Xu
ee9a5d7611 feat: introduce FlowRouteValue (#4263) 
* feat: introduce `FlowRouteKey` and `FlowRouteValue`

* feat: put `FlowRouteValue` values in flow creation

* feat: use `FlowRouteValue`

* refactor: remove `PeerLookupServiceRef` in `DdlContext`

* chore: remove unused code

* Update src/common/meta/src/key.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* chore: apply suggestions from CR

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
 2024-07-03 19:46:16 +00:00
Yohan Wal
c4db9e8aa7 fix!: forbid to change information_schema (#4233) 
* fix: forbid to change tables in information_schema

* refactor: use unified read-only check function

* test: add more sqlness tests for information_schema

* refactor: move is_readonly_schema to common_catalog
 2024-07-03 03:09:23 +00:00
Ning Sun
11cf9c827e feat: dbeaver mysql compatibility, use statement and information_schema.tables (#4218) 
* feat: add more placeholder field in information_schema.tables

* feat: make schema modifiable for use statement

* chore: add todo items

* fix: resolve lint issues after data type changes

* chore: update sqlness results

* refactor: patch for select database is no longer needed

* test: align tests and data types

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* fix: use canonicalize_identifier for database name

* feat: add all columns for information_schema.tables

* test: remove vairables from sqlness results

* feat: add to_string impl for table options

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-03 01:31:13 +00:00
tison
2665616f72 build(deps): Upgrade OpenDAL to 0.47 (#4224) 
* catch up changes

Signed-off-by: tison <wander4096@gmail.com>

* fmt

Signed-off-by: tison <wander4096@gmail.com>

* Fix cache for 0471 (#7)

* Fix cache for 0471

Signed-off-by: Xuanwo <github@xuanwo.io>

* Make clippy happy

Signed-off-by: Xuanwo <github@xuanwo.io>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

* use opendal's exported type

Signed-off-by: tison <wander4096@gmail.com>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* fmt

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2024-07-01 17:05:15 +00:00
Ruihang Xia
a4e99f5666 feat: basic implement of matches fn (#4222) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/common/function/src/scalars/matches.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update src/common/function/src/scalars/matches.rs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* revert typo fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore typo unqualifed

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enhance grammar restrictions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

* todo about tokenizer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reverse order

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rewrite escape_pattern

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-07-01 07:59:59 +00:00
Weny Xu
6a634f8e5d feat: register & deregister region failure detectors actively (#4223) 
* feat: Use DATANODE_LEASE_SECS from distributed_time_constants for heartbeat pause duration

* feat: introduce `RegionFailureDetectorController` to manage region failure detectors

* feat: add `RegionFailureDetectorController` to `DdlContext`

* feat: add `region_failure_detector_controller` to `Context` in region migration

* feat: register region failure detectors during rollback region migration procedure

* feat: deregister region failure detectors during drop table procedure

* feat: register region failure detectors during create table procedure

* fix: update meta config

* chore: apply suggestions from CR

* chore: avoid cloning

* chore: rename

* chore: reduce the size of the test

* chore: apply suggestions from CR

* chore: move channel initialization into `RegionSupervisor::channel`

* chore: minor refactor

* chore: rename ident
 2024-07-01 05:58:27 +00:00
Weny Xu
8cbe7166b0 refactor: migrate region failover implementation to region migration (#4172) 
* refactor: migrate region failover implementation to region migration

* fix: use HEARTBEAT_INTERVAL_MILLIS as lease secs

* fix: return false if leader is downgraded

* fix: only remove failure detector after submitting procedure successfully

* feat: ignore dropped region

* refactor: retrieve table routes in batches

* refactor: disable region failover on local WAL implementation

* fix: move the guard into procedure

* feat: use real peer addr

* feat: use interval instead of sleep

* chore: rename `HeartbeatSender` to `HeartbeatAcceptor`

* chore: apply suggestions from CR

* chore: reduce duplicate code

* chore: apply suggestions from CR

* feat: lookup peer addr

* chore: add comments

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-06-25 11:58:17 +00:00
Weny Xu
b2f61aa1cf fix: format error correctly (#4204) 
* chore: remove TODO comments

* fix: format error correctly
 2024-06-25 07:56:13 +00:00
discord9
a44fe627ce feat: heartbeat task&peer lookup in proc (#4179) 
* feat: herat beat task

* feat: use real flow peer allocator when building

* feat: add peer look up in ddl context

* fix: drop flow test

* refactor: per review(WIP)

* refactor: not check if is alive

* refactor: per review

* refactor: remove useless `reset`

* refactor: per bot advices

* refactor: alive peer

* chore: bot review
 2024-06-24 15:06:33 +00:00
Ruihang Xia
fce65c97e3 feat: make RegionScanner aware of PartitionRange (#4170) 
* define PartitionRange

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add optimizer rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl aggr stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add fallback method

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document and rename struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-21 09:54:22 +00:00
Ruihang Xia
21c89f3247 perf: optimize RecordBatch to HttpOutput conversion (#4178) 
* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 70ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add profiler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 50ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 160ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix windows build

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-20 12:33:58 +00:00
dennis zhuang
4306cba866 feat: show database options (#4174) 
* test: test create table with database ttl

* feat: show database options

* fix: comment

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: fix CR comments and refactor

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-06-20 04:21:58 +00:00
Jeremyhi
4c3d4af127 feat: register flow node (#4166) 
* feat: rename keys.rs to key.rs

* feat: refactor datanode keys

* feat: add flownode key

* feat: keep flownode's lease info in metasrv

* feat: flow selector

* feat: impl_try_from_lease_key and impl_from_str_lease_key to simple code
 2024-06-20 03:46:19 +00:00
LFC
cc2f7efb98 chore: bump datafusion version to fix last_value regression (#4169) 
* chore: bump datafusion version to fix `last_value` regression

* fix: resolve PR comments

* fix ci
 2024-06-19 07:47:17 +00:00
Weny Xu
ea2d067cf1 feat: implement the OrderedBatchProducer (#4134) 
* feat: implement the `OrderedBatchProducer`

* test: add test of cancel safety

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: simplify the `BackgroundProducerWorker`

* feat: implement the OrderedBatchProducer v2

* refactor: switch to `OrderedBatchProducer`

* chore: rename to `MAX_FLUSH_QUEUE_SIZE`

* refactor: switch to `OrderedBatchProducerV2`

* refactor: remove `OrderedBatchProducerV1`

* test: add tests

* refactor: make config configurable

* refactor: minor refactor

* chore: remove unused code

* chore: remove `benchmarks` crate

* chore: update config doc

* chore: remove unused comment

* refactor: refactor client registry

* refactor: rename `max_batch_size` to `max_batch_bytes`

* chore: use constant value

* chore: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: remove the `CHANNEL_SIZE`

* chore: apply suggestions from CR

* fix: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-06-18 07:20:01 +00:00
Ning Sun
70d113a355 feat: update default size of bgworkers, add hbworkers (#4129) 
* feat: update default size of bgworkers, add hbworkers

* feat: update frontend heartbeat as well

* chore: update sample config files and default settings

* chore: update config docs

* Revert "chore: update config docs"

This reverts commit 8107f4c120.

* Revert "chore: update sample config files and default settings"

This reverts commit f5ae701c8d.

* feat: use default heartbeat runtime size

* chore: update config docs
 2024-06-18 06:18:37 +00:00
Jeremyhi
141d017576 feat: enable metasrv to accept flownode's heartbeats (#4160) 
* feat: make metasrv can accept flownode's heartbeat

* chore: proto
 2024-06-18 04:07:46 +00:00
Lei, HUANG
f8ec46493f refactor: simplify parquet writer (#4112) 
* refactor: simplify parquet writer

* chore: fix clippy

* refactor: use AsyncArrowWriter instead of BufferedWriter

* refactor: remove BufferedWriter

* fix: add chunk parameter to avoid entity too small issue

* refactor: use AtomicUsize instead of Mutex

* fix: add chunk argument to stream_to_parquet

* chore: fmt

* wip: fail check

* fix: check

* fmt

* refactor: use impl Future instead of async_trait

* fmt

* refactor: use associate types
 2024-06-13 07:32:47 +00:00
Yingwen
65f8b72d34 feat: Implement RegionScanner for SeqScan (#4060) 
* feat: ordered builder wip

* feat: impl RegionScanner for SeqScan

* feat: implement scan_partition and build_stream

* chore: return SeqScan as RegionScanner

* fix: group parts

* feat: split parts

* chore: reader metrics

* chore: metrics

* chore: remove unused codes

* chore: support holding a group of ranges in ScanPart

* feat: group ScanParts to ScanParts

* feat: impl SeqScanner again

* chore: observe build cost in ScannerMetrics

* chore: fix compiler warnings

* style: fix clippy

* docs: update config docs

* chore: forward DisplayAs to scanner

* test: update sqlness tests

* chore: update debug fmt

* chore: custom debug for timestamp

fix test compiling issue with common-macro when running
cargo nextest -p common-time

* chore: update debug format

* feat: update fmt for scan part

* chore: fix warning

* fix: sanitize parallelism

* feat: split parts

* test: fix config api test

* feat: update logs

* chore: Revert "chore: remove unused codes"

This reverts commit b548b30a01eeded59b1a0a8d89f9293ca63afc41.

* chore: Revert "docs: update config docs"

This reverts commit a7997e78d6ddcf635560574de8c1948c495bdd12.

* feat: each partition scan files in parallel

* test: fix config api test

* docs: fix typo

* chore: address comments, simplify tests

* feat: global semaphore

* feat: always spawn task

* chore: simplify default explain output format

* handle output partiton number is 0

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-12 08:21:30 +00:00
LFC
5a6021e34f refactor: remove substrait ser/de for region query in standalone (#3812) 
* refactor: remove substrait serde for region query in standalone

* fix ci

* move QueryRequest to common-query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: format toml

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-11 09:33:47 +00:00