rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-27 18:30:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,907 Commits 137 Branches 254 Tags
81d1b31d65d4e574baf71be7895f2909b20162ff
Commit Graph

69 Commits

Author SHA1 Message Date
Niwaka
06d273b75a fix: GET: /config return all configurations when running standalone (#2630) 
* fix: config api return all configurations when running standalone

* chore: follow same style

* fix: avoid panic
 2023-11-08 03:22:19 +00:00
dennis zhuang
ff15bc41d6 feat: improve object storage cache (#2522) 
* feat: refactor object storage cache with moka

* chore: minor fixes

* fix: concurrent issues and invalidate cache after write/delete

* chore: minor changes

* fix: cargo lock

* refactor: rename

* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib

* fix: typo

* chore: style

* fix: toml format

* chore: toml

* fix: toml format

* Update src/object-store/src/layers/lru_cache/read_cache.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: update src/object-store/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: refactor and apply suggestions

* fix: typo

* feat: adds back allow list for caching

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* refactor: wrap inner Accessor with Arc

* chore: remove run_pending_task in read and write path

* chore: the arc is unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-10-08 03:27:49 +00:00
LinFeng
621c6f371b feat: limit grpc message size (#2459) 
* feat: add two grpc config options

Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size

* test: add integration tests for message size limit
 2023-09-22 02:07:46 +00:00
Ruihang Xia
34d6288945 feat: bring back sqlness and integration tests (#2448) 
* enable integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* disable sqlness region failover

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enable sqlness in CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort unstable result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* set require_lease_before_startup to true

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix inconsistent cache

* replace windows path chars

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore some integration cases in windows

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "ignore some integration cases in windows"

This reverts commit 122478b7c1.

* disable windows for now

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix close region bug in RegionHeartbeatResponseHandler

* disable failover tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-20 09:17:30 +00:00
Zhenchi
3a39215f11 feat: migrate file engine from table to reigon (#2365) 
* feat: migrate file engine from table to reigon

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/file-engine/src/engine.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: specify ts index for file engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: handle time index for external table

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: some integration testsg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add file schema and table schema compatibility

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: compatible file schema to region schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: simplify close

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: implement set_writable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: tests-integration compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local>
 2023-09-18 08:02:43 +00:00
LFC
4b13c88752 fix: resolve more integration tests (#2406) 
* fix: resolve more integration tests

* Update tests-integration/tests/http.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2023-09-15 09:43:16 +00:00
LFC
d08b05c963 fix: make test-integration able to compile (#2384) 
* fix: make test-integration able to compile

* chore: fmt toml

---------

Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-13 12:42:46 +00:00
Ruihang Xia
718246ea1a feat: implement heartbeat for region server (#2279) 
* retrieve region stats from region server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement heartbeat handler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* start datanode with region server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* disable non-unit test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement heartbeat task

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
18250c4803 feat: implement Flight and gRPC services for RegionServer (#2226) 
* extract FlightCraft trait

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* split service handler in GrpcServer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* left grpc server implement

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* start region server if configured

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-22 13:30:09 +00:00
Ruihang Xia
4aaf6aa51b feat: implement query API for RegionServer (#2197) 
* some initial change

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl dummy structs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* decode and send logical plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement table scan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add some comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-17 11:02:31 +00:00
sh2
202540823f refactor!: move prometheus routes to default http server (#2005) 
* move prometheus routes to default http server

Signed-off-by: sh2 <shawnhxh@outlook.com>

* fix ci test and remove the server logic of prometheus

* remove unused import and prometheus relevant code

* fix ci: rustfmt and test

* fix ci: silly fmt

* fix ci: silly silly fmt

* change `/prom_store` back to `/prometheus`

* remove unused variable

---------

Signed-off-by: sh2 <shawnhxh@outlook.com>
 2023-08-16 03:21:14 +00:00
shuiyisong
7f51141ed0 refactor: auth crate (#2148) 
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
 2023-08-14 02:51:26 +00:00
dennis zhuang
6f40128058 feat!: enable telemetry by default (#2137) 
* feat: remove greptimedb-telemetry feature

* feat: adds enable_telemetry option to metasrv and datanode

* refactor: move data_home from file config to storage config

* feat: store the installation uuid into datanode and metasrv working home

* fix: cargo toml fmt

* test: ignore region failver test when using local fle storage

* test: ignore telemetry reporter in test mode

* feat: print warning log when enabling telemetry

* chore: the telemetry doc link

* chore: remove enable_telemetry from datanode example config file

* refactor: rename GREPTIMEDB_TELEMETRY_CLIENT_REQUEST_TIMEOUT

* chore: rename print_warn_log to print_anonymous_usage_data_disclaimer
 2023-08-11 14:50:40 +00:00
shuiyisong
5eb2c609a3 fix: auth in grpc (#2056) 
* fix: auth in grpc

* fix: change to return err

* fix: add grpc test

* fix: add http test

* fix: add mysql and pg test
 2023-08-01 15:18:31 +00:00
LFC
4fdb6d2f21 refactor: remote catalog uses memory (#1926) 
* refactor: remote catalog uses memory

* rebase develop

* fix: resolve PR comments
 2023-07-12 09:33:33 +00:00
Eugene Tolbakov
674bfd85c7 chore(prom)!: rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus (#1931) 
* chore(prom): rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus

* chore: apply clippy suggestions

* chore: adjust format according to rustfmt
 2023-07-12 14:47:09 +08:00
Weny Xu
2d30f4c373 fix: fix broken CI (#1933) 2023-07-11 14:48:41 +08:00
Eugene Tolbakov
fc850c9988 feat(config-endpoint): add initial implementation (#1896) 
* feat(config-endpoint): add initial implementation

* feat: add initial handler implementation

* fix: apply clippy suggestions, use axum response instead of string

* feat: address CR suggestions

* fix: minor adjustments in formatting

* fix: add a test

* feat: add to_toml_string method to options

* fix: adjust the assertion for the integration test

* fix: adjust expected indents

* fix: adjust assertion for the integration test

* fix: improve according to clippy
 2023-07-11 11:08:32 +08:00
Niwaka
66e5ed5483 feat: support gcs storage (#1781) 2023-07-05 23:03:51 +08:00
Lei, HUANG
f287d3115b chore: replace result assertions (#1840) 
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g

* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
 2023-06-27 19:14:48 +08:00
LFC
fcff66e039 chore: deny unused results (#1825) 
* chore: deny unused results

* rebase
 2023-06-27 15:33:53 +08:00
Weny Xu
dcfce49cff refactor(datanode): move Instance heartbeat task to Datanode struct (#1832) 
* refactor(datanode): move Instance heartbeat to Datanode struct

* chore: apply suggestions from CR

* fix: start heartbeat task after instance starts
 2023-06-27 12:32:20 +08:00
Ruihang Xia
a95f8767a8 refactor: merge catalog provider & schema provider into catalog manager (#1803) 
* move  to expr_factory

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move configs into service_config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move GrpcQueryHandler into distributed.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile and test in catalog sub-crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix table-procedure compile and test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix query compile and tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix datanode compile and tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix catalog/query/script/servers compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix frontend compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix nextest except information_schema

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* support information_schema

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove other structs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change deregister_table's return type to empty tuple

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-26 15:08:59 +08:00
Niwaka
a103614fd2 feat: support /api/v1/series for Prometheus (#1620) 
* feat: support /api/v1/series for Prometheus

* chore: error handling

* feat: update tests
 2023-06-06 10:29:16 +08:00
LFC
5004cf6d9a feat: make grpc insert requests in a batch (#1687) 
* feat: make Prometheus remote write in a batch

* rebase

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-06-02 09:06:48 +00:00
Ning Sun
e7a410573b test: fix sqlx compatibility and adds integration test for sqlx (#1686) 
* test: fix sqlx compatibility and adds integration test for sqlx

* test: correct insert statements
 2023-06-01 15:43:13 +08:00
dennis zhuang
ae8203fafa fix: prepare statement doesn't support insert clause (#1680) 
* fix: insert clause doesn't support prepare statement

* fix: manifeste dir

* fix: format

* fix: temp path
 2023-05-31 20:14:58 +08:00
Weny Xu
0460f3ae30 test: add write test for region failover (#1673) 
* test: add write test for region failover

* test: add written data assertion after failover

* test: support more storage types
 2023-05-31 15:42:00 +08:00
Zou Wei
ae81c7329d feat: support azblob storage. (#1659) 
* feat:support azblob storage.

* test:add some tests.

* refactor:use if-let.
 2023-05-30 19:59:38 +08:00
dennis zhuang
7c55783e53 feat!: reorganize the storage layout (#1609) 
* feat: adds data_home to DataOptions

* refactor: split out object store stuffs from datanode instance

* feat: move data_home into FileConfig

* refactor: object storage layers

* feat: adds datanode path to procedure paths

* feat: temp commit

* refactor: clean code

* fix: forgot files

* fix: forgot files

* Update src/common/test-util/src/ports.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: compile error

* chore: cr comments

* fix: dependencies order in cargo

* fix: data path in test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-23 13:58:26 +08:00
LFC
f64527da22 feat: region failover procedure (#1558) 
* feat: region failover procedure
 2023-05-22 19:54:52 +08:00
localhost
34c7f78861 chore: add configurator to http server (#1488) 
* chore: add configurator params to start server fun

* chore: update plugins type

---------

Co-authored-by: paomian <qtang@greptime.com>
 2023-05-08 10:55:03 +00:00
Vanish
6aae5b7286 feat: prevent sensitive information (key, password, secrets etc.) from being printed in plain (#1501) 
* feat: add secret type

* chore: replace key, password, secrets with secret type.

* chore: use secrecy

* chore: remove redundant file

* style: taplo fmt
 2023-05-01 20:54:54 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489) 
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
 2023-04-28 22:12:57 +08:00
Yingwen
b1f7ad097a test: Fix s3 region in test (#1493) 2023-04-27 12:25:20 +00:00
Weny Xu
a77a4a4bd1 fix: add s3 region info (#1492) 
fix: add region info
 2023-04-27 19:13:01 +08:00
Lei, HUANG
fb9978e95d refactor: catalog (#1454) 
* wip

* add schema_async

* remove CatalogList

* remove catalog provider and schema provider

* fix

* fix: rename table

* fix: sqlness

* fix: ignore tonic error metadata

* fix: table engine name

* feat: rename catalog_async to catalog

* respect engine name in table regional value when deregistering tables

* fix: CR
 2023-04-26 08:36:40 +00:00
dennis zhuang
8d8a480dc1 fix: object store caching bug, #1466 (#1467) 
* fix: object store caching bug, #1466

* fix: forgot to add S3WithCache tests
 2023-04-25 21:48:51 +08:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
Ruihang Xia
2cf828da3c feat: implement Prometheus-compatible API in gRPC (#1449) 
* update greptime-proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove duplicate delete enum

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl handler and service

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-24 18:03:48 +08:00
Weny Xu
451f9d2d4e feat: support multi table engines (#1277) 
* feat: support multi table engines

* refactor: adapt SqlHandler to support multiple table engines

* refactor: refactor TableEngineManager

* chore: apply review suggestions

* chore: apply review suggestions

* chore: apply review suggestions

* chore: snafu context styling
 2023-04-03 14:49:12 +00:00
LFC
48c2841e4d feat: execute python script in distributed mode (#1264) 
* feat: execute python script in distributed mode

* fix: rebase develop
 2023-04-02 20:36:48 +08:00
localhost
a2b262ebc0 chore: add http metrics server in datanode node when greptime start in distributed mode (#1256) 
* chore: add http metrics server in datanode node when greptime start in distributed mode

* chore: add some docs and license

* chore: change metrics_addr to resolve address already in use error

* chore add metrics for meta service

* chore: replace metrics exporter http server from hyper to axum

* chore: format

* fix: datanode mode branching error

* fix: sqlness test address already in use and start metrics in defualt config

* chore: change metrics location

* chore: use builder pattern to builder httpserver

* chore: remove useless debug_assert macro in httpserver builder

* chore: resolve conflicting build error

* chore: format code
 2023-03-31 18:37:52 +08:00
dennis zhuang
563adbabe9 feat!: improve region manifest service (#1268) 
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-31 10:42:00 +08:00
Lei, HUANG
5edd2a3dbe feat: upgrade opendal (#1245) 
* chore: upgrade opendal

* chore: finish upgrading opendal

* fix: clippy complaints

* fix some tests

* fix: all unit tests

* chore: rebase develop

* fix: sqlness tests

* optimize imports

* chore: rebase develop

* doc: add todo
 2023-03-28 09:47:33 +08:00
Lei, HUANG
17eb99bc52 feat: allow manual table flush through HTTP API (#1184) 2023-03-15 20:15:34 +08:00
LFC
604c20a83d refactor: remove the SQL execution interfaces in Datanode (#1135) 
* refactor: remove the SQL execution interfaces in Datanode
 2023-03-13 18:45:30 +08:00
Eugene Tolbakov
b31a6cb506 refactor: replace tempdir with tempfile (#1123) 
* refactor: replace tempdir with tempfile

* refactor(query): move tempfile dependency under the workspace's Cargo.toml

* refactor(tempfile): create common-test-util

* refactor(tempfile): fix toml format

* refactor(tempfile): remove tempfile out of dependencies

* refactor(tempfile): fix incorrect toml
 2023-03-08 11:15:56 +08:00
yuanbohan
d4e0dc3685 feat: specify prom server start addr (#1111) 
* feat: specify promql server start addr

* refactor: rename promql to prom in Prometheus API server scenario
 2023-03-06 11:07:21 +08:00
Xuanwo
fdd17c6eeb refactor: Clean up re-export of opendal services (#1067) 
Signed-off-by: Xuanwo <github@xuanwo.io>
 2023-02-23 14:12:34 +08:00