rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-16 18:22:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,108 Commits 106 Branches 227 Tags
7437820bdc866865d5f0f2e33683f79efba8366d
Commit Graph

120 Commits

Author SHA1 Message Date
Yingwen
115d9eea8d chore: Log version and arguments (#1744) 2023-06-09 11:38:08 +08:00
Lei, HUANG
8cda1635cc feat: make jemalloc the default allocator (#1733) 
* feat: add jemalloc metrics

* fix: dep format
 2023-06-06 12:11:22 +00:00
Ruihang Xia
2615718999 feat: merge scan for distributed execution (#1660) 
* generate exec plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move DatanodeClients to client crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wip MergeScanExec::to_stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix default catalog

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix expand order of new stage

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move sqlness cases contains plan out of common dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor information schema to allow duplicated scan call

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: ignore two cases due to substrait

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reorganise sqlness common cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typos

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* redact round robin partition number

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

* skip tranforming projection

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert common/order

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/query/src/dist_plan/merge_scan.rs

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore region failover IT

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result again and again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* unignore some tests about projection

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enable failover tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-06-02 06:42:54 +00:00
Yingwen
548f0d1e2a feat: Add app version metric (#1685) 
* feat: Add app version metric

* chore: use greptimedb instead of greptime
 2023-06-01 14:31:08 +08:00
Yingwen
9d179802b8 feat: Add a global TTL option for all tables (#1679) 
* feat: Add a global TTL option for all tables

* docs: update config examples

* chore: print start command and options when standalone/frontend starts
 2023-05-31 15:36:25 +08:00
Zou Wei
ae81c7329d feat: support azblob storage. (#1659) 
* feat:support azblob storage.

* test:add some tests.

* refactor:use if-let.
 2023-05-30 19:59:38 +08:00
fys
fa4a497d75 feat: add cache for catalog kv backend (#1592) 
* feat: add kvbackend cache

* fix: cargo fmt
 2023-05-24 15:07:29 +08:00
Ning Sun
ddca0307d1 feat: more configurable logging levels (#1630) 
* feat: make logging level more configurable

* chore: resolve lint warnings

* fix: correct default level for h2

* chore: update text copy
 2023-05-24 14:47:41 +08:00
dennis zhuang
7c55783e53 feat!: reorganize the storage layout (#1609) 
* feat: adds data_home to DataOptions

* refactor: split out object store stuffs from datanode instance

* feat: move data_home into FileConfig

* refactor: object storage layers

* feat: adds datanode path to procedure paths

* feat: temp commit

* refactor: clean code

* fix: forgot files

* fix: forgot files

* Update src/common/test-util/src/ports.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: compile error

* chore: cr comments

* fix: dependencies order in cargo

* fix: data path in test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-23 13:58:26 +08:00
Weny Xu
e646490d16 chore: fix code styling (#1623) 2023-05-23 10:09:34 +08:00
Lei, HUANG
8e7ec4626b refactor: remove useless error (#1624) 
* refactor: remove useless

* fix: remove useless error variant
 2023-05-22 22:55:27 +08:00
LFC
f64527da22 feat: region failover procedure (#1558) 
* feat: region failover procedure
 2023-05-22 19:54:52 +08:00
Ruihang Xia
e5aad0f607 feat: distributed planner basic (#1599) 
* basic skeleton

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change QueryEngineState's constructor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* install extension planner

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tidy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-22 11:48:03 +08:00
Ruihang Xia
c3eeda7d84 refactor(frontend): adjust code structure (#1615) 
* move  to expr_factory

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move configs into service_config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move GrpcQueryHandler into distributed.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-20 02:09:20 +08:00
zyy17
4920836021 refactor: support parsing env list (#1595) 
* refactor: support parse env list

* refactor: set 'multiple = true' for metasrv_addr cli option and remove duplicated parsing
 2023-05-17 14:37:08 +08:00
WU Jingdi
fb1ac0cb9c feat: support user config manifest compression (#1579) 
* feat: support user config manifest compression

* chore: change style

* chore: enhance test
 2023-05-16 11:02:59 +08:00
zyy17
4fc173acf0 refactor: support layered configuration (#1535) 
* refactor: add a layered configuration by using config-rs

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add 'env_var_prefix' for 'load_options()' and remove duplicate default construction in frontend

* refactor: add test_config_precedence_order in standalone

* refactor: add 'test_config_precedence_order()' test case in metasrv

* refactor: add 'test_config_precedence_order()' test case in datanode

* refactor: refine the naming '*_env_var_*' -> '*_env_vars_*'

* refactor: fix clippy error

* refactor: refine error naming 'LoadConfig' -> 'LoadLayeredConfig' and add Location

* refactor: move 'env_vars_prefix' to clap options

* fix: use '__' as envrionment variables separator and simplify load_layered_options()

* refactor: derive 'Default' for StartCommand and use default function to simplify the test cases

* fix: clippy error

* chore: update comments

* chore(deps): update deps info

* refactor(naming): 'env_vars_prefix' -> 'env_prefix'

* refactor: simplify the code

* refactor: change some argument type of 'load_layered_options()'

* refactor: simplify the code

* refactor: remove unnecessary 'clone()'

* refactor: add 'GREPTIMEDB_*' prefix for env_prefix

* refactor: modify configuration precedence order: cli > config file > environment variables > default values

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2023-05-13 22:37:47 +08:00
LFC
8fef32f8ef feat: enable tokio console in cluster mode (#1512) 
* feat: enable tokio console subscriber

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-05-11 14:35:06 +08:00
localhost
34c7f78861 chore: add configurator to http server (#1488) 
* chore: add configurator params to start server fun

* chore: update plugins type

---------

Co-authored-by: paomian <qtang@greptime.com>
 2023-05-08 10:55:03 +00:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501) 
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
 2023-05-01 20:54:54 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489) 
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
 2023-04-28 22:12:57 +08:00
Lei, HUANG
9e4887f29f fix: disable dashboard (#1494) 2023-04-27 22:55:15 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444) 
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
 2023-04-27 15:30:04 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454) 
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
 2023-04-26 08:36:40 +00:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
Lei, HUANG
92c0808766 fix: frontend opt should respect http addr in config file when no com… (#1456) 
* fix: frontend opt should respect http addr in config file when no command options is given

* refactor: command line options should be Option<bool>

* fix: ci
 2023-04-25 03:43:42 +00:00
Ning Sun
c5dba29f9e refactor: remove redundant plugins argument (#1436) 2023-04-23 12:39:46 +08:00
zyy17
554a69ea54 refactor: add disable_dashboard option and disable dashboard in metasrv and datanode (#1343) 
* refactor: add disable_dashboard option and disable dashboard in metasrv and datanode

* refactor: skip disable_dashboard filed in toml file

* refactor: simplify the http initialization
 2023-04-07 16:45:25 +08:00
Ruihang Xia
da66138e80 refactor(error): remove backtrace, and introduce call-site location for debugging (#1329) 
* wip: global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unneeded tests of errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ErrorExt trait implementator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-06 04:06:00 +00:00
dennis zhuang
bf8c717022 feat: try to do manifest checkpoint on opening region (#1321) 2023-04-04 21:36:54 +08:00
LFC
48c2841e4d feat: execute python script in distributed mode (#1264) 
* feat: execute python script in distributed mode

* fix: rebase develop
 2023-04-02 20:36:48 +08:00
Lei, HUANG
0253136333 feat: buffered parquet writer (#1263) 
* wip: use

* rebase develop

* chore: fix typos

* feat: replace export parquet writer with buffered writer

* fix: some cr comments

* feat: add sst_write_buffer_size config item to config how many bytes to buffer before flush to underlying storage

* chore: reabse onto develop
 2023-04-01 17:21:19 +08:00
localhost
a2b262ebc0 chore: add http metrics server in datanode node when greptime start in distributed mode (#1256) 
* chore: add http metrics server in datanode node when greptime start in distributed mode

* chore: add some docs and license

* chore: change metrics_addr to resolve address already in use error

* chore add metrics for meta service

* chore: replace metrics exporter http server from hyper to axum

* chore: format

* fix: datanode mode branching error

* fix: sqlness test address already in use and start metrics in defualt config

* chore: change metrics location

* chore: use builder pattern to builder httpserver

* chore: remove useless debug_assert macro in httpserver builder

* chore: resolve conflicting build error

* chore: format code
 2023-03-31 18:37:52 +08:00
dennis zhuang
563adbabe9 feat!: improve region manifest service (#1268) 
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-31 10:42:00 +08:00
Ruihang Xia
f351ee7042 docs: update document string and site (#1211) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-03-21 07:01:08 +00:00
Weny Xu
7cfa30b2ab feat: add shutdown for standalone and metasrv (#1174) 2023-03-17 11:35:17 +08:00
Xieqijun
cd8be77968 feat(procedure): Max retry time (#1095) 
* feat: procedure config

* fix: modify config

* feat: add retry logic

* feat: add error

* feat: add it

* feat: add it

* feat: add it

* feat: rm retry from runner

* feat: use backon

* feat: add retry_interval

* feat: add retry_interval

* fix: conflict

* fix: cr

* feat: add retry error and id

* feat: rename

* refactor: execute

* feat: use config dir

* fix: cr

* fix: cr

* fix: fmt

* fix: fmt

* fix: pr

* fix: it

* fix: rm unless cmd params

* feat: add toml

* fix: ut

* feat: add rolling back

* fix: cr

* fix: cr

* fix: cr

* fix: ci

* fix: ci

* fix: ci

* chore: Apply suggestions from code review

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-15 08:28:08 +00:00
Weny Xu
cdc111b607 refactor: make the cmd hold the application instance (#1159) 2023-03-14 15:18:50 +08:00
LFC
604c20a83d refactor: remove the SQL execution interfaces in Datanode (#1135) 
* refactor: remove the SQL execution interfaces in Datanode
 2023-03-13 18:45:30 +08:00
Weny Xu
c7f114c8fa feat: add shutdown for frontend (#1161) 2023-03-13 17:59:36 +08:00
Eugene Tolbakov
b31a6cb506 refactor: replace tempdir with tempfile (#1123) 
* refactor: replace tempdir with tempfile

* refactor(query): move tempfile dependency under the workspace's Cargo.toml

* refactor(tempfile): create common-test-util

* refactor(tempfile): fix toml format

* refactor(tempfile): remove tempfile out of dependencies

* refactor(tempfile): fix incorrect toml
 2023-03-08 11:15:56 +08:00
Lei, HUANG
a4c01f4a3a feat: memory profiling (#1124) 
* feat: use jemalloc as default allocator

* feat: add feature for mem-prof

* feat: add errors

* make common-mem-prof optional dep

* fix: toml format

* doc: add profile doc

* fix: typo
 2023-03-07 17:12:51 +08:00
yuanbohan
d4e0dc3685 feat: specify prom server start addr (#1111) 
* feat: specify promql server start addr

* refactor: rename promql to prom in Prometheus API server scenario
 2023-03-06 11:07:21 +08:00
Yingwen
f1b65d9b77 test: fix datanode::test_read_from_config_file (#1106) 
* test: Fix datanode::test_read_from_config_file

* test: frontend and metasrv don't read example toml file
 2023-03-01 18:31:40 +08:00
LFC
8cd69f441e feat: REPL issues logical plan to DB (#1097) 2023-02-28 16:59:48 +08:00
Yingwen
bd377ef329 feat: Procedure to create table and register table to catalog (#1040) 
* feat: Add table-procedures crate

* feat: Implement procedure to create table

* feat: Integrate procedure manager to datanode

* test: Test CreateTableProcedure

* refactor: Rename table-procedures to table-procedure

* feat: Implement create_table_by_procedure

* chore: Remove comment

* chore: Add todo

* feat: Add procedure config to standalone mode

* feat: Register table-procedure loaders

* feat: Address review comments

CreateTableProcedure just return error if the subprocedure is failed

* chore: Address CR comments
 2023-02-27 11:49:23 +08:00
LFC
df751c38b4 feat: a simple REPL for debugging purpose (#1048) 
* feat: a simple REPL for debugging purpose

* fix: rebase develop
 2023-02-27 11:00:15 +08:00
Yingwen
f6e871708a chore: Rename MetaClientOpts to MetaClientOptions (#1075) 
* fix: Serialize FrontendOptions to toml

* fix: Serialize DatanodeOptions to toml

* fix: Serialize StandaloneOptions to toml

See https://users.rust-lang.org/t/why-toml-to-string-get-error-valueaftertable/85903/2

* chore!: Rename MetaClientOpts to MetaClientOptions

BREAKING CHANGE: Change the meta_client_opts in the config file to
meta_client_options
 2023-02-24 16:28:38 +08:00
Lei, HUANG
2df8143ad5 feat: support table ttl (#1052) 
* feat: purge expired sst on compaction

* chore: add more log

* fix: clippy

* fix: mark expired ssts as compacting before picking candidates

* fix: some CR comments

* fix: remove useless result

* fix: cr comments
 2023-02-22 16:56:20 +08:00
Lei, HUANG
af1f8d6101 feat: file purger (#1030) 
* wip

* wip

* feat: file purger

* chore: add tests

* feat: delete removed file on sst merge

* chore: move MockAccessLayer to test_util

* fix: some cr comments

* feat: add await termination for scheduler

* fix: some cr comments

* chore: rename max_file_in_level0 to max_files_in_level0
 2023-02-19 14:56:41 +08:00