rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-08 06:12:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,020 Commits 103 Branches 226 Tags
e5a215de465234fbff2558ea4ea78e1efe7b9904
Commit Graph

331 Commits

Author SHA1 Message Date
Ruihang Xia
e5aad0f607 feat: distributed planner basic (#1599) 
* basic skeleton

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change QueryEngineState's constructor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* install extension planner

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tidy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-22 11:48:03 +08:00
localhost
a6ec79ee30 chore: add a uniform prefix to the metrics using the official recommendation of (#1590) 2023-05-17 11:08:49 +08:00
WU Jingdi
fb1ac0cb9c feat: support user config manifest compression (#1579) 
* feat: support user config manifest compression

* chore: change style

* chore: enhance test
 2023-05-16 11:02:59 +08:00
fys
027707d969 feat: support frontend-meta heartbeat (#1555) 
* feat: support frontend heartbeat

* fix: typo "reponse" -> "response"

* add ut

* enable start heartbeat task

* chore: frontend id is specified by metasrv, not in the frontend startup parameter

* fix typo

* self-cr

* cr

* cr

* cr

* remove unnecessary headers

* use the member id in the header as the node id
 2023-05-15 09:54:45 +00:00
LFC
4ae0b5e185 test: move instances tests to "tests-integration" (#1573) 
* test: move standalone and distributed instances tests from "frontend" crate to "tests-integration"

* fix: resolve PR comments
 2023-05-15 12:00:43 +08:00
zyy17
4fc173acf0 refactor: support layered configuration (#1535) 
* refactor: add a layered configuration by using config-rs

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add 'env_var_prefix' for 'load_options()' and remove duplicate default construction in frontend

* refactor: add test_config_precedence_order in standalone

* refactor: add 'test_config_precedence_order()' test case in metasrv

* refactor: add 'test_config_precedence_order()' test case in datanode

* refactor: refine the naming '*_env_var_*' -> '*_env_vars_*'

* refactor: fix clippy error

* refactor: refine error naming 'LoadConfig' -> 'LoadLayeredConfig' and add Location

* refactor: move 'env_vars_prefix' to clap options

* fix: use '__' as envrionment variables separator and simplify load_layered_options()

* refactor: derive 'Default' for StartCommand and use default function to simplify the test cases

* fix: clippy error

* chore: update comments

* chore(deps): update deps info

* refactor(naming): 'env_vars_prefix' -> 'env_prefix'

* refactor: simplify the code

* refactor: change some argument type of 'load_layered_options()'

* refactor: simplify the code

* refactor: remove unnecessary 'clone()'

* refactor: add 'GREPTIMEDB_*' prefix for env_prefix

* refactor: modify configuration precedence order: cli > config file > environment variables > default values

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2023-05-13 22:37:47 +08:00
LFC
8fef32f8ef feat: enable tokio console in cluster mode (#1512) 
* feat: enable tokio console subscriber

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-05-11 14:35:06 +08:00
Weny Xu
44aef6fcbd feat(datanode): iImplement the heartbeat response handler (#1547) 
* feat(datanode): implement instruction handler

* chore: apply suggestion from CR

* refactor: refactor heartbeat response handler
 2023-05-11 09:27:13 +08:00
WU Jingdi
486bb2ee8e feat: Compress manifest and checkpoint (#1497) 
* feat: Compress manifest and checkpoint

* refactor: use file extention infer compression type

* chore: apply suggestions from CR

* Update src/storage/src/manifest/storage.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: CR advices

* chore: Fix bugs, strengthen test

* chore: Fix CR, strengthen test

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-10 07:53:06 +00:00
Weny Xu
020c55e260 refactor: change mailbox_messages to mailbox_message (#1557) 2023-05-10 07:17:11 +00:00
Yingwen
ee3e1dbdaa feat: Use LocalScheduler framework to implement FlushScheduler (#1531) 
* test: simplify countdownlatch

* feat: impl Drop for LocalScheduler

* feat(storage): Impl FlushRequest and FlushHandler

* feat(storage): Use scheduler to handle flush job

* chore(storage): remove unused code

* feat(storage): Use new type pattern for RegionMap

* feat(storage): Remove on_success callback

* feat(storage): Address CR comments and add some metrics to flush
 2023-05-10 07:16:51 +00:00
Weny Xu
fbb7db42aa chore: unify code styling (#1523) 2023-05-10 11:10:39 +08:00
Ning Sun
b0ab641602 feat: add catalog/schema/table count as catalog metrics (#1499) 
* feat: add catalog/schema/table count as catalog metrics

* test: add integration tests for catalog metrics
 2023-05-05 05:54:12 +00:00
dennis zhuang
b1920c41a4 fix: object store cache bug (#1482) 
* feat: use streaming read instead of reading whole file

* feat: enable atomic writing for object store file caching

* fix: recover existing keys from local cache

* test: recovering keys from local file cache for LruCachePolicy

* Update src/datanode/src/instance.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr comments

* feat: md5 hash caching path

* fix: test

* fix: read cache

* Update src/object-store/src/cache_policy.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-04 18:25:40 +08:00
JeremyHi
d7a906e0bd feat: metasrv mailbox (#1481) 
* refactor: id first in pusher_key

* feat: is_acceptable for multi roles

* feat: mailbox

* fix: channel for mailbox

* feat: impl mailbox via heartbeat

* chore: add unit test for mailbox

* chore: by cr

* chore: typo

* chore: refactor the mailbox API

* chore: br cr

* chore: check timeout interval to 10ms

* chore: add response header
 2023-05-04 15:42:43 +08:00
Ning Sun
494ad570c5 feat: update pgwire to 0.14 (#1504) 2023-05-04 14:24:26 +08:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501) 
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
 2023-05-01 20:54:54 +08:00
yuanbohan
cca34aa914 chore: upgrade promql-parser version (#1484) 2023-04-27 13:10:15 +00:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444) 
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
 2023-04-27 15:30:04 +08:00
Ruihang Xia
939a51aea9 feat: adopt REPLACE interceptor and quit all processes on exit (#1478) 
* bump version and update test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* quit all processes on drop

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update tests/runner/src/env.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-04-27 07:16:41 +00:00
Weny Xu
bf35620904 refactor: refactor BufferedWriter (#1439) 
* feat: implement ApproximateBufWriter

* refactor: refactor BufferedWriter

* refactor: remove ApproximateBufWriter

* fix: fix losing pending writes issue

* chore: fmt

* chore: remove unused import

* chore: rename method name

* feat: return written row count

* chore: apply suggestions from CR

* fix: fix counting the bytes_written twice issue
 2023-04-27 14:45:33 +08:00
dennis zhuang
b88d8e5b82 feat: bump opendal to 0.33 (#1479) 2023-04-27 12:13:18 +08:00
discord9
ef4e473e6d fix: recompile&register scripts as UDF on reboot (#1421) 
* fixme: recompile somewhere else

* feat: re-compile&re-register all scripts in table

* fix: allow empty scripts table

* chore: add non-blocking somewhere

* chore: PR advices

* chore: more PR advices

* style: remove useless join

* style: remove redunent code

* refactor: use `bg` runtime instead

* style: cargo fmt
 2023-04-26 16:30:58 +08:00
Ning Sun
1a245f35b9 feat: improve metrics and log level (#1470) 
* refactor: tune log and metrics for meta/frontend

* feat: add panic counter
 2023-04-26 13:13:40 +08:00
Ruihang Xia
4d9afee8ef chore(deps): update substrait dep in client (#1453) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-25 16:21:59 +08:00
Ruihang Xia
f9ea6b63bf feat: impl instant query and add tests (#1452) 
* feat: impl instant query and add tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-25 11:08:14 +08:00
Weny Xu
f2167663b2 feat: support to create external table (#1372) 
* feat: support to create external table

* chore: apply suggestions from CR

* test: add create external table without ts type

* chore: apply suggestions from CR

* fix: fix import typo

* refactor: move consts to table crate

* chore: apply suggestions from CR

* refactor: rename create_table_schema
 2023-04-24 14:43:12 +08:00
LFC
17daf4cdff feat: support "delete" in distributed mode (#1441) 
* feat: support "delete" in distributed mode

* fix: resolve PR comments
 2023-04-24 12:07:50 +08:00
shuiyisong
7c6754d03e feat: meter write request (#1447) 
* chore: add write meter

* chore: update meter macro

* chore: update meter framework url to https
 2023-04-24 11:42:06 +08:00
Weny Xu
22b5a94d02 feat: support creating the physical plan for JSON and CSV files (#1424) 
* feat: support creating the physical plan for JSON and CSV files

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor(file-table-engine): use datasource Format instead
 2023-04-24 10:17:11 +08:00
dennis zhuang
2a9f482bc7 feat: show create table (#1336) 
* temp commit

* feat: impl Display for CreateTable statement

* feat: impl show create table for standalone

* fix: forgot show.rs

* feat: clean code

* fix: typo

* feat: impl show create table for distributed

* test: add show create table sqlness test

* fix: typo

* fix: sqlness tests

* feat: render partition rules for distributed table

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/sql/src/statements/create.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* fix: compile error

* fix: missing column comments and extra table options

* test: add show create table test

* test: add show create table test

* chore: timestamp precision

* fix: test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-04-21 11:37:16 +08:00
Yingwen
9cd2cf630d feat: procedures for file table engine (#1417) 
* refactor: Add table_ref() to requests as their methods

* feat: Add CreateImmutableFileTable

* feat: Add DropImmutableFileTable

* feat: Implement TableEngineProcedure for ImmutableFileTableEngine

* feat: Add common-procedure-test crate

* refactor: mito engine use common-procedure-test to test procedures

* test: Add test for create and drop table

* chore: Address review comments
 2023-04-20 18:52:44 +08:00
Lei, HUANG
0e4d4f0300 chore: release 0.2.0 (#1413) 
* chore: bump version to v0.2.0

* chore: bump dashboard to v0.2.1

* chore: remove push uhub step

* fix: static assets path prefix
 2023-04-18 22:12:13 +08:00
shuiyisong
145f8eb5a7 refactor: parallelize open table (#1392) 
* refactor: change open_table to parallel on datanode startup

* chore: try move out register schema table

* chore: change mito engine to key lock

* chore: minor change

* chore: minor change

* chore: update error definition

* chore: remove rwlock on tables

* chore: try parallel register table on schema provider

* chore: add rt log

* chore: add region open rt log

* chore: add actual open region rt log

* chore: add recover rt log

* chore: divide to three part rt log

* chore: remove debug log

* chore: add replay rt log

* chore: update cargo lock

* chore: remove debug log

* chore: revert unused change

* chore: update err msg

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fix cr issue

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fix cr issue

* chore: fix cr issue

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-04-18 21:36:29 +08:00
discord9
de8b889701 chore: update RustPython depend (#1406) 
* chore: update RustPython to newer version

* chore: bump ver

* chore: PR advices
 2023-04-18 15:39:57 +08:00
Near
c6f024a171 feat: Add metrics for cache hit/miss for object store cache (#1405) 
* Add the cache hit/miss counter

* Verify the cache metrics are included

* Resolve comments

* Rename the error kind label name to be consistent with other metrics

* Rename the object store metric names

* Avoid using glob imports

* Format the code

* chore: Update src/object-store/src/metrics.rs mod doc

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-04-18 14:08:19 +08:00
localhost
0c88bb09e3 chore: add some metrics for grpc client (#1398) 
* chore: add some metrics for grpc client

* chore: add grpc preix and change metrics-exporter-ptometheus to add global prefix

---------

Co-authored-by: paomian <qtang@greptime.com>
 2023-04-18 13:55:01 +08:00
Ruihang Xia
f4190cfca6 fix: table scan without projection (#1404) 
* fix: table scan without projection

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update PR reference

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-17 20:10:36 +08:00
Ruihang Xia
dc85a4b5bb feat: migrate substrait to datafusion official implementation (#1238) 
* some test cases will fail

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert version changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update substrait-proto version

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update df again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/common/substrait/Cargo.toml

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* revert COPY FROM / COPY TO sqlness to standalone only

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-04-17 15:54:35 +08:00
Weny Xu
408de51be8 feat: implement JsonOpener and CsvOpener (#1367) 
* feat: introduce JsonOpener and CsvOpener

* refactor: refactor Opener

* docs: add doc
 2023-04-17 11:42:16 +08:00
LFC
f7b7a9c801 feat: implement COPY for cluster (#1388) 2023-04-17 11:04:47 +08:00
Ruihang Xia
a6e41cdd7b chore: bump arrow, parquet, datafusion and tonic (#1386) 
* bump arrow, parquet, datafusion, tonic and greptime-proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add analyzer and fix test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-15 00:03:15 +08:00
Ning Sun
90cd3bb5c9 chore: switch mysql_async to git dep (#1383) 2023-04-14 07:04:34 +00:00
Ning Sun
43391e0162 chore: update pgwire and rustls libraries (#1380) 
* feat: update pgwire to 0.13 and fix grafana compatibility

* chore: update pgwire and rustls

* chore: remove unsued imports

* style: format toml
 2023-04-14 11:06:01 +08:00
Yingwen
0819582a26 feat: Add alter table procedure (#1354) 
* feat: Implement AlterTableProcedure

* test: Test alter table procedure

* feat: support alter table by procedure in datanode

* chore: update comment
 2023-04-13 14:05:53 +08:00
Lei, HUANG
76640402ba fix: update cargo lock (#1375) 2023-04-13 11:08:35 +08:00
Weny Xu
fac9c17a9b feat: implement infer schema from single file (#1348) 
* feat: implement infer schema from file

* feat: implement compression type

* refactor: remove unnecessary BufReader

* refactor: remove SyncIoBridge and using tokio_util::io::SyncIoBridge instead

* chore: apply suggestions from CR
 2023-04-11 16:59:30 +08:00
Weny Xu
a90798a2c1 test: add tests for file table engine (#1353) 
* test: add tests for file table engine

* test: refactor open table test and add close engine test
 2023-04-11 06:25:08 +00:00
Weny Xu
29c6155ae3 feat: introduce file table engine (#1323) 
* feat: introduce file table engine

* chore: apply cr suggestions

* refactor: refactor immutable manifest

* chore: apply cr suggestions

* refactor: refactor immutable manifest

* chore: apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: apply suggestions from CR

* chore: apply suggestions from code review

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-04-10 12:03:36 +08:00
Lei, HUANG
b7bdee6de9 feat: ignoring time zone info when import from external files (#1341) 
* feat: ignore timezone info when copy from external files

* chore: rebase onto develop
 2023-04-10 11:41:34 +08:00