rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-24 17:00:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,026 Commits 131 Branches 253 Tags
8e7ec4626bb9d809ca0ecbb1d9bac9a9349b78f0
Commit Graph

297 Commits

Author SHA1 Message Date
Lei, HUANG
8e7ec4626b refactor: remove useless error (#1624) 
* refactor: remove useless

* fix: remove useless error variant
 2023-05-22 22:55:27 +08:00
LFC
f64527da22 feat: region failover procedure (#1558) 
* feat: region failover procedure
 2023-05-22 19:54:52 +08:00
Yingwen
6dbceb1ad5 feat: Trigger flush based on global write buffer size (#1585) 
* feat(storage): Add AllocTracker

* feat(storage): flush request wip

* feat(storage): support global write buffer size

* fix(storage): Test and fix size based strategy

* test(storage): Test AllocTracker

* test(storage): Test pick_by_write_buffer_full

* docs: Add flush config example

* test(storage): Test schedule_engine_flush

* feat(storage): Add metrics for write buffer size

* chore(flush): Add log when triggering flush by global buffer

* chore(storage): track allocation in update_stats
 2023-05-22 19:00:30 +08:00
Ruihang Xia
e5aad0f607 feat: distributed planner basic (#1599) 
* basic skeleton

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change QueryEngineState's constructor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* install extension planner

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tidy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-22 11:48:03 +08:00
zyy17
4920836021 refactor: support parsing env list (#1595) 
* refactor: support parse env list

* refactor: set 'multiple = true' for metasrv_addr cli option and remove duplicated parsing
 2023-05-17 14:37:08 +08:00
WU Jingdi
fb1ac0cb9c feat: support user config manifest compression (#1579) 
* feat: support user config manifest compression

* chore: change style

* chore: enhance test
 2023-05-16 11:02:59 +08:00
Niwaka
856ab5bea7 feat: make RepeatedTask invoke remove_outdated_meta method (#1578) 
* feat: make RepeatedTask invoke remove_outdated_meta method

* fix: typo

* chore: improve error message
 2023-05-16 10:21:35 +08:00
Eugene Tolbakov
122bd5f0ab feat(tql): add initial implementation for explain & analyze (#1427) 
* feat(tql): resolve conflicts after merge,formatting and clippy issues, add sqlness tests, adjust explain with start, end, step

* feat(tql): adjust sqlness assertions
 2023-05-16 07:28:24 +08:00
fys
027707d969 feat: support frontend-meta heartbeat (#1555) 
* feat: support frontend heartbeat

* fix: typo "reponse" -> "response"

* add ut

* enable start heartbeat task

* chore: frontend id is specified by metasrv, not in the frontend startup parameter

* fix typo

* self-cr

* cr

* cr

* cr

* remove unnecessary headers

* use the member id in the header as the node id
 2023-05-15 09:54:45 +00:00
Yingwen
8d54d40b21 feat: Add FlushPicker to flush regions periodically (#1559) 
* feat: Add FlushPicker

* feat(storage): Add close to StorageEngine

* style(storage): fix clippy

* feat(storage): Close regions in StorageEngine::close

* chore(storage): Clear requests on scheduler stop

* test(storage): Test flush picker

* feat(storage): Add metrics for auto flush

* feat(storage): Add flush reason and record it in metrics

* feat: Expose flush config

docs(config): Update config example

* refactor(storage): Run auto flush task in FlushScheduler

* refactor(storage): Add FlushItem trait to make FlushPicker easy to test
 2023-05-15 17:29:28 +08:00
Weny Xu
66903d42e1 feat: implement OpenTableHandler (#1567) 
* feat: implement OpenTableHandler

* chore: apply suggestion from CR

* chore: apply suggestion from CR
 2023-05-15 10:47:28 +08:00
LFC
8fef32f8ef feat: enable tokio console in cluster mode (#1512) 
* feat: enable tokio console subscriber

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-05-11 14:35:06 +08:00
Weny Xu
44aef6fcbd feat(datanode): iImplement the heartbeat response handler (#1547) 
* feat(datanode): implement instruction handler

* chore: apply suggestion from CR

* refactor: refactor heartbeat response handler
 2023-05-11 09:27:13 +08:00
dennis zhuang
b1920c41a4 fix: object store cache bug (#1482) 
* feat: use streaming read instead of reading whole file

* feat: enable atomic writing for object store file caching

* fix: recover existing keys from local cache

* test: recovering keys from local file cache for LruCachePolicy

* Update src/datanode/src/instance.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr comments

* feat: md5 hash caching path

* fix: test

* fix: read cache

* Update src/object-store/src/cache_policy.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-04 18:25:40 +08:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501) 
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
 2023-05-01 20:54:54 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489) 
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
 2023-04-28 22:12:57 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444) 
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
 2023-04-27 15:30:04 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454) 
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
 2023-04-26 08:36:40 +00:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
Ruihang Xia
2cf828da3c feat: implement Prometheus-compatible API in gRPC (#1449) 
* update greptime-proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove duplicate delete enum

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl handler and service

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-24 18:03:48 +08:00
Weny Xu
f2167663b2 feat: support to create external table (#1372) 
* feat: support to create external table

* chore: apply suggestions from CR

* test: add create external table without ts type

* chore: apply suggestions from CR

* fix: fix import typo

* refactor: move consts to table crate

* chore: apply suggestions from CR

* refactor: rename create_table_schema
 2023-04-24 14:43:12 +08:00
LFC
17daf4cdff feat: support "delete" in distributed mode (#1441) 
* feat: support "delete" in distributed mode

* fix: resolve PR comments
 2023-04-24 12:07:50 +08:00
Yingwen
1ee9ad4ca1 feat: manage multiple engine procedure in the engine manager (#1434) 
* feat(table): Add engine procedure to engine manager

* feat(datanode): Get engine procedure from engine manager

* feat(table-procedure): Add source error to SubprocedureFailed

* test: Enable procedure in tests and pass all tests

* style(table-procedure): Fix clippy
 2023-04-23 10:04:09 +08:00
dennis zhuang
2a9f482bc7 feat: show create table (#1336) 
* temp commit

* feat: impl Display for CreateTable statement

* feat: impl show create table for standalone

* fix: forgot show.rs

* feat: clean code

* fix: typo

* feat: impl show create table for distributed

* test: add show create table sqlness test

* fix: typo

* fix: sqlness tests

* feat: render partition rules for distributed table

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements/create.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* fix: compile error

* fix: missing column comments and extra table options

* test: add show create table test

* test: add show create table test

* chore: timestamp precision

* fix: test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-04-21 11:37:16 +08:00
Weny Xu
d5e4662181 refactor: refactor stmt_options_to_table_options (#1403) 
refactor: move stmt_options_to_table_options to query crate
 2023-04-21 11:08:01 +08:00
Yingwen
830367b8f4 feat: Drop table by procedure (#1401) 
* feat: Add drop table procedure

* feat: support dropping table by procedure on datanode

* test: Add test for DropTableProcedure

* test: Test drop table by procedure

* chore: update comments

* fix: Make on_remove_from_catalog idempotent
 2023-04-20 15:57:56 +08:00
localhost
0c88bb09e3 chore: add some metrics for grpc client (#1398) 
* chore: add some metrics for grpc client

* chore: add grpc preix and change metrics-exporter-ptometheus to add global prefix

---------

Co-authored-by: paomian <qtang@greptime.com>
 2023-04-18 13:55:01 +08:00
LFC
f7b7a9c801 feat: implement COPY for cluster (#1388) 2023-04-17 11:04:47 +08:00
Ruihang Xia
a6e41cdd7b chore: bump arrow, parquet, datafusion and tonic (#1386) 
* bump arrow, parquet, datafusion, tonic and greptime-proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add analyzer and fix test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-15 00:03:15 +08:00
Lei, HUANG
68e64a6ce9 feat: add some metrics (#1384) 
* feat: add some metrics

* fix: compile errors
 2023-04-14 20:46:45 +08:00
Ruihang Xia
3e7f7e3e8d fix: compile error in develop branch (#1376) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-13 15:19:00 +08:00
Yingwen
0819582a26 feat: Add alter table procedure (#1354) 
* feat: Implement AlterTableProcedure

* test: Test alter table procedure

* feat: support alter table by procedure in datanode

* chore: update comment
 2023-04-13 14:05:53 +08:00
LFC
33dbf7264f refactor: unify the execution of show stmt (#1340) 
* refactor: unify the execution of show stmt
 2023-04-12 23:09:07 +08:00
localhost
1a21a6ea41 chore: set metasrv and datanode heartbeat log level to trace (#1357) 2023-04-11 11:21:29 +08:00
Lei, HUANG
b7bdee6de9 feat: ignoring time zone info when import from external files (#1341) 
* feat: ignore timezone info when copy from external files

* chore: rebase onto develop
 2023-04-10 11:41:34 +08:00
Lei, HUANG
c850e9695a fix: stream inserts when copying from external file (#1338) 
* fix: stream inserts when copying from external file

* fix: reset pending bytes once insertion succeeds

* Update src/datanode/src/sql/copy_table_from.rs

Co-authored-by: LFC <bayinamine@gmail.com>

---------

Co-authored-by: LFC <bayinamine@gmail.com>
 2023-04-10 10:44:12 +08:00
LFC
f8b6a6b219 fix!: not allowed to create column name same with keyword without quoted (#1333) 
* fix: not allowed to create column name same with keyword without quoted

* fix: tests

* Update src/sql/src/parsers/create_parser.rs

Co-authored-by: Ning Sun <classicning@gmail.com>

* fix: tests

---------

Co-authored-by: Ning Sun <classicning@gmail.com>
 2023-04-06 15:34:26 +08:00
Ruihang Xia
da66138e80 refactor(error): remove backtrace, and introduce call-site location for debugging (#1329) 
* wip: global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unneeded tests of errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ErrorExt trait implementator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-06 04:06:00 +00:00
dennis zhuang
bf8c717022 feat: try to do manifest checkpoint on opening region (#1321) 2023-04-04 21:36:54 +08:00
Weny Xu
ef134479ef feat: support multi table engines in distributed mode (#1316) 
* chore: bump greptime-proto to 59afacd

* feat: support multi table engines in distributed mode
 2023-04-04 10:27:08 +08:00
Weny Xu
451f9d2d4e feat: support multi table engines (#1277) 
* feat: support multi table engines

* refactor: adapt SqlHandler to support multiple table engines

* refactor: refactor TableEngineManager

* chore: apply review suggestions

* chore: apply review suggestions

* chore: apply review suggestions

* chore: snafu context styling
 2023-04-03 14:49:12 +00:00
LFC
215cea151f refactor: move PromQL execution to Frontend (#1297) 
* refactor: move PromQL execution to Frontend
 2023-04-03 11:34:03 +08:00
LFC
48c2841e4d feat: execute python script in distributed mode (#1264) 
* feat: execute python script in distributed mode

* fix: rebase develop
 2023-04-02 20:36:48 +08:00
Lei, HUANG
0253136333 feat: buffered parquet writer (#1263) 
* wip: use

* rebase develop

* chore: fix typos

* feat: replace export parquet writer with buffered writer

* fix: some cr comments

* feat: add sst_write_buffer_size config item to config how many bytes to buffer before flush to underlying storage

* chore: reabse onto develop
 2023-04-01 17:21:19 +08:00
localhost
a2b262ebc0 chore: add http metrics server in datanode node when greptime start in distributed mode (#1256) 
* chore: add http metrics server in datanode node when greptime start in distributed mode

* chore: add some docs and license

* chore: change metrics_addr to resolve address already in use error

* chore add metrics for meta service

* chore: replace metrics exporter http server from hyper to axum

* chore: format

* fix: datanode mode branching error

* fix: sqlness test address already in use and start metrics in defualt config

* chore: change metrics location

* chore: use builder pattern to builder httpserver

* chore: remove useless debug_assert macro in httpserver builder

* chore: resolve conflicting build error

* chore: format code
 2023-03-31 18:37:52 +08:00
dennis zhuang
972f64c3d7 chore: improve opendal layers (#1295) 
* chore: improve opendal layers

* chore: log level
 2023-03-31 09:48:11 +00:00
LFC
eb77f9aafd feat: start LocalManager in Metasrv (#1279) 
* feat: procedure store in Metasrv, backed by Etcd; start `LocalManager` in Metasrv leader

* fix: resolve PR comments

* fix: resolve PR comments
 2023-03-31 15:32:59 +08:00
Yingwen
dee20144d7 feat: Implement procedure to alter a table for mito engine (#1259) 
* feat: wip

* fix: Fix CreateMitoTable::table_schema not initialized from json

* feat: Implement AlterMitoTable procedure

* test: Add test for alter procedure

* feat: Register alter procedure

* fix: Recover procedures after catalog manager is started

* feat: Simplify usage of table schema in create table procedure

* test: Add rename test

* test: Add drop columns test
 2023-03-31 14:40:54 +08:00
dennis zhuang
563adbabe9 feat!: improve region manifest service (#1268) 
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-31 10:42:00 +08:00
LFC
fae293310c feat: unify describe table execution (#1285) 2023-03-31 09:59:19 +08:00