rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-06-01 04:40:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,495 Commits 139 Branches 254 Tags
8e9f2ffce42e2aa4fed1da166c0a649ed31c450d
Commit Graph

159 Commits

Author SHA1 Message Date
LFC
f64527da22 feat: region failover procedure (#1558) 
* feat: region failover procedure
 2023-05-22 19:54:52 +08:00
Ruihang Xia
e5aad0f607 feat: distributed planner basic (#1599) 
* basic skeleton

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change QueryEngineState's constructor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* install extension planner

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tidy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-22 11:48:03 +08:00
Ruihang Xia
c3eeda7d84 refactor(frontend): adjust code structure (#1615) 
* move  to expr_factory

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move configs into service_config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move GrpcQueryHandler into distributed.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-20 02:09:20 +08:00
zyy17
4920836021 refactor: support parsing env list (#1595) 
* refactor: support parse env list

* refactor: set 'multiple = true' for metasrv_addr cli option and remove duplicated parsing
 2023-05-17 14:37:08 +08:00
WU Jingdi
fb1ac0cb9c feat: support user config manifest compression (#1579) 
* feat: support user config manifest compression

* chore: change style

* chore: enhance test
 2023-05-16 11:02:59 +08:00
zyy17
4fc173acf0 refactor: support layered configuration (#1535) 
* refactor: add a layered configuration by using config-rs

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add 'env_var_prefix' for 'load_options()' and remove duplicate default construction in frontend

* refactor: add test_config_precedence_order in standalone

* refactor: add 'test_config_precedence_order()' test case in metasrv

* refactor: add 'test_config_precedence_order()' test case in datanode

* refactor: refine the naming '*_env_var_*' -> '*_env_vars_*'

* refactor: fix clippy error

* refactor: refine error naming 'LoadConfig' -> 'LoadLayeredConfig' and add Location

* refactor: move 'env_vars_prefix' to clap options

* fix: use '__' as envrionment variables separator and simplify load_layered_options()

* refactor: derive 'Default' for StartCommand and use default function to simplify the test cases

* fix: clippy error

* chore: update comments

* chore(deps): update deps info

* refactor(naming): 'env_vars_prefix' -> 'env_prefix'

* refactor: simplify the code

* refactor: change some argument type of 'load_layered_options()'

* refactor: simplify the code

* refactor: remove unnecessary 'clone()'

* refactor: add 'GREPTIMEDB_*' prefix for env_prefix

* refactor: modify configuration precedence order: cli > config file > environment variables > default values

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2023-05-13 22:37:47 +08:00
LFC
8fef32f8ef feat: enable tokio console in cluster mode (#1512) 
* feat: enable tokio console subscriber

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-05-11 14:35:06 +08:00
localhost
34c7f78861 chore: add configurator to http server (#1488) 
* chore: add configurator params to start server fun

* chore: update plugins type

---------

Co-authored-by: paomian <qtang@greptime.com>
 2023-05-08 10:55:03 +00:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501) 
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
 2023-05-01 20:54:54 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489) 
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
 2023-04-28 22:12:57 +08:00
Lei, HUANG
9e4887f29f fix: disable dashboard (#1494) 2023-04-27 22:55:15 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444) 
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
 2023-04-27 15:30:04 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454) 
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
 2023-04-26 08:36:40 +00:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
Lei, HUANG
92c0808766 fix: frontend opt should respect http addr in config file when no com… (#1456) 
* fix: frontend opt should respect http addr in config file when no command options is given

* refactor: command line options should be Option<bool>

* fix: ci
 2023-04-25 03:43:42 +00:00
Ning Sun
c5dba29f9e refactor: remove redundant plugins argument (#1436) 2023-04-23 12:39:46 +08:00
zyy17
554a69ea54 refactor: add disable_dashboard option and disable dashboard in metasrv and datanode (#1343) 
* refactor: add disable_dashboard option and disable dashboard in metasrv and datanode

* refactor: skip disable_dashboard filed in toml file

* refactor: simplify the http initialization
 2023-04-07 16:45:25 +08:00
Ruihang Xia
da66138e80 refactor(error): remove backtrace, and introduce call-site location for debugging (#1329) 
* wip: global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unneeded tests of errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ErrorExt trait implementator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-06 04:06:00 +00:00
dennis zhuang
bf8c717022 feat: try to do manifest checkpoint on opening region (#1321) 2023-04-04 21:36:54 +08:00
LFC
48c2841e4d feat: execute python script in distributed mode (#1264) 
* feat: execute python script in distributed mode

* fix: rebase develop
 2023-04-02 20:36:48 +08:00
Lei, HUANG
0253136333 feat: buffered parquet writer (#1263) 
* wip: use

* rebase develop

* chore: fix typos

* feat: replace export parquet writer with buffered writer

* fix: some cr comments

* feat: add sst_write_buffer_size config item to config how many bytes to buffer before flush to underlying storage

* chore: reabse onto develop
 2023-04-01 17:21:19 +08:00
localhost
a2b262ebc0 chore: add http metrics server in datanode node when greptime start in distributed mode (#1256) 
* chore: add http metrics server in datanode node when greptime start in distributed mode

* chore: add some docs and license

* chore: change metrics_addr to resolve address already in use error

* chore add metrics for meta service

* chore: replace metrics exporter http server from hyper to axum

* chore: format

* fix: datanode mode branching error

* fix: sqlness test address already in use and start metrics in defualt config

* chore: change metrics location

* chore: use builder pattern to builder httpserver

* chore: remove useless debug_assert macro in httpserver builder

* chore: resolve conflicting build error

* chore: format code
 2023-03-31 18:37:52 +08:00
dennis zhuang
563adbabe9 feat!: improve region manifest service (#1268) 
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-31 10:42:00 +08:00
Ruihang Xia
f351ee7042 docs: update document string and site (#1211) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-03-21 07:01:08 +00:00
Weny Xu
7cfa30b2ab feat: add shutdown for standalone and metasrv (#1174) 2023-03-17 11:35:17 +08:00
Xieqijun
cd8be77968 feat(procedure): Max retry time (#1095) 
* feat: procedure config

* fix: modify config

* feat: add retry logic

* feat: add error

* feat: add it

* feat: add it

* feat: add it

* feat: rm retry from runner

* feat: use backon

* feat: add retry_interval

* feat: add retry_interval

* fix: conflict

* fix: cr

* feat: add retry error and id

* feat: rename

* refactor: execute

* feat: use config dir

* fix: cr

* fix: cr

* fix: fmt

* fix: fmt

* fix: pr

* fix: it

* fix: rm unless cmd params

* feat: add toml

* fix: ut

* feat: add rolling back

* fix: cr

* fix: cr

* fix: cr

* fix: ci

* fix: ci

* fix: ci

* chore: Apply suggestions from code review

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-15 08:28:08 +00:00
Weny Xu
cdc111b607 refactor: make the cmd hold the application instance (#1159) 2023-03-14 15:18:50 +08:00
LFC
604c20a83d refactor: remove the SQL execution interfaces in Datanode (#1135) 
* refactor: remove the SQL execution interfaces in Datanode
 2023-03-13 18:45:30 +08:00
Weny Xu
c7f114c8fa feat: add shutdown for frontend (#1161) 2023-03-13 17:59:36 +08:00
Eugene Tolbakov
b31a6cb506 refactor: replace tempdir with tempfile (#1123) 
* refactor: replace tempdir with tempfile

* refactor(query): move tempfile dependency under the workspace's Cargo.toml

* refactor(tempfile): create common-test-util

* refactor(tempfile): fix toml format

* refactor(tempfile): remove tempfile out of dependencies

* refactor(tempfile): fix incorrect toml
 2023-03-08 11:15:56 +08:00
Lei, HUANG
a4c01f4a3a feat: memory profiling (#1124) 
* feat: use jemalloc as default allocator

* feat: add feature for mem-prof

* feat: add errors

* make common-mem-prof optional dep

* fix: toml format

* doc: add profile doc

* fix: typo
 2023-03-07 17:12:51 +08:00
yuanbohan
d4e0dc3685 feat: specify prom server start addr (#1111) 
* feat: specify promql server start addr

* refactor: rename promql to prom in Prometheus API server scenario
 2023-03-06 11:07:21 +08:00
Yingwen
f1b65d9b77 test: fix datanode::test_read_from_config_file (#1106) 
* test: Fix datanode::test_read_from_config_file

* test: frontend and metasrv don't read example toml file
 2023-03-01 18:31:40 +08:00
LFC
8cd69f441e feat: REPL issues logical plan to DB (#1097) 2023-02-28 16:59:48 +08:00
Yingwen
bd377ef329 feat: Procedure to create table and register table to catalog (#1040) 
* feat: Add table-procedures crate

* feat: Implement procedure to create table

* feat: Integrate procedure manager to datanode

* test: Test CreateTableProcedure

* refactor: Rename table-procedures to table-procedure

* feat: Implement create_table_by_procedure

* chore: Remove comment

* chore: Add todo

* feat: Add procedure config to standalone mode

* feat: Register table-procedure loaders

* feat: Address review comments

CreateTableProcedure just return error if the subprocedure is failed

* chore: Address CR comments
 2023-02-27 11:49:23 +08:00
LFC
df751c38b4 feat: a simple REPL for debugging purpose (#1048) 
* feat: a simple REPL for debugging purpose

* fix: rebase develop
 2023-02-27 11:00:15 +08:00
Yingwen
f6e871708a chore: Rename MetaClientOpts to MetaClientOptions (#1075) 
* fix: Serialize FrontendOptions to toml

* fix: Serialize DatanodeOptions to toml

* fix: Serialize StandaloneOptions to toml

See https://users.rust-lang.org/t/why-toml-to-string-get-error-valueaftertable/85903/2

* chore!: Rename MetaClientOpts to MetaClientOptions

BREAKING CHANGE: Change the meta_client_opts in the config file to
meta_client_options
 2023-02-24 16:28:38 +08:00
Lei, HUANG
2df8143ad5 feat: support table ttl (#1052) 
* feat: purge expired sst on compaction

* chore: add more log

* fix: clippy

* fix: mark expired ssts as compacting before picking candidates

* fix: some CR comments

* fix: remove useless result

* fix: cr comments
 2023-02-22 16:56:20 +08:00
Lei, HUANG
af1f8d6101 feat: file purger (#1030) 
* wip

* wip

* feat: file purger

* chore: add tests

* feat: delete removed file on sst merge

* chore: move MockAccessLayer to test_util

* fix: some cr comments

* feat: add await termination for scheduler

* fix: some cr comments

* chore: rename max_file_in_level0 to max_files_in_level0
 2023-02-19 14:56:41 +08:00
Lei, HUANG
75b8afe043 feat: compaction integration (#997) 
* feat: trigger compaction on flush

* chore: rebase develop

* feat: add config item max_file_in_level0 and remove compaction_after_flush

* fix: cr comments

* chore: add unit test to cover Timestamp::new_inclusive

* fix: workaround to fix future is not Sync

* fix: future is not sync

* fix: some cr comments
 2023-02-15 14:14:07 +08:00
shuiyisong
9989a8c192 fix: check full table name during logical plan creation (#948) 2023-02-09 17:23:28 +08:00
Ruihang Xia
fc9276c79d feat: export promql service in server (#924) 
* chore: some tiny typo/style fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: add promql server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* works for mocked query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* integration test case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* expose promql api to our http server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust router structure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-02-03 08:28:56 +00:00
LFC
af935671b2 feat: support "use" in GRPC requests (#922) 
* feat: support "use catalog and schema"(behave like the "use" in MySQL) in GRPC requests

* fix: rebase develop
 2023-02-02 20:02:56 +08:00
Yun Chen
a7dc86ffe5 feat: oss storage support (#911) 
* feat: add oss storage support

* fix: ci build format check

* fix: align OSS to Oss

* fix: cr comments

* fix: rename OSS to Oss in integration tests

* fix: clippy fix
 2023-01-29 20:09:38 +08:00
shuiyisong
aafc26c788 feat: add mysql reject_no_database (#896) 
* chore: update opensrv-mysql to main

* refactor: change mysql server struct

* feat: add option to reject no database mysql connection request

* chore: remove unused condition

* chore: rebase develop

* chore: make reject_no_database optional
 2023-01-29 04:09:47 +00:00
Ruihang Xia
36a13dafb7 build(deps): bump tokio to 1.24.2 (#900) 
deps: bump tokio to 1.24.2

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-01-29 11:13:37 +08:00
shuiyisong
6960739b3d feat: add authorize to UserProvider trait (#879) 
* feat: add SchemaValidator

* feat: add schema validator to mysql shim

* chore: pass schema validator to http auth layer

* feat: add schema validator to http

* feat: add schema validator to pg

* feat: add schema validator to pg

* feat: add schema validator test

* chore: remove println in test

* chore: use !matches

* refactor: refac authenticate and authorize in http auth

* refactor: refac authenticate and authorize in http auth

* chore: typo

* chore: minor change

* refactor: merge schema_validator into user_providier

* chore: fix license issue

* refactor: change http query param from database to db

* chore: fix cr issue
 2023-01-18 12:42:08 +08:00
Ning Sun
ecb71f81be feat: add --rpc-hostname option to datanode for a persist address to store in meta (#871) 
* feat: add --rpc-hostname option

* fix: config file and hostname parsing

* Apply suggestions from code review

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>
 2023-01-17 10:50:50 +08:00
fys
6f5639fccd feat: add load_based selector in meta (#874) 
* fix: wrong error info

* add derive hash for StatKey

* add a attrs field in Context

* add load_based selector

* add license

* make Nodestat module public

* add meta startup config item about selector

* cr: remove attrs, add concrete type in context

* cr: change region_number type to Option<u64>

* cr: add comment in example.toml

* cr
 2023-01-17 10:25:00 +08:00
Lei, HUANG
90fcaa8487 feat: expose wal config (#852) 
* feat: wal config

* fix: use human-readable string in wal config

* feat: copy ReadableSize and humanize size config items in toml files

* fix: clippy
 2023-01-10 16:07:26 +08:00