rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2025-12-25 23:49:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,936 Commits 103 Branches 223 Tags
script_wrapper
Commit Graph

80 Commits

Author SHA1 Message Date
Yingwen
abbac46c05 fix: do not expose manifest compression algorithm (#2835) 
* fix: don't allow to set manifest compression algorithm

* docs: update config examples
 2023-11-29 05:49:40 +00:00
WU Jingdi
0badb3715e feat: support sample ratio in trace (#2809) 
* feat: support sample ratio in trace

* chore: fix code advice
 2023-11-27 06:46:46 +00:00
Yingwen
b9146c88ff refactor: Remove usages of the old storage crate (#2777) 
* chore: remove storage from some crate

* feat: remove storage config

* feat: remove storage from cmd

* feat: impl stream_to_parquet

* feat: remove storage from operator

* feat: remove stream writer from mito2

* feat: remove storage from project toml

* test: fix config api test

* docs: remove outdated configs

* refactor: remove storage directory
 2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f feat(mito): implements row group level page cache (#2688) 
* feat: add page cache

* docs: update mito config toml

* feat: impl CachedPageReader

* feat: use cache reader to read row group

* feat: do not fetch data if we have pages in cache

* chore: return if nothing to fetch

* feat: enlarge page cache size

* test: test write read parquet

* test: test cache

* docs: update comments

* test: fix config api test

* feat: cache metrics

* feat: change default page cache size

* test: fix config api test
 2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40 feat: Enable distributed tracing in greptimedb (#2755) 
* feat: implement distributed tracing

* fix: change usage of span

* fix: use otlp as exporter

* chore: update dependence

* chore: add span info

* chore: add alias

* chore: use instrument instead of trace
 2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074 fix!: improve user experience on setting compression type (#2765) 
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type

* Update src/common/datasource/src/compression.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-11-17 06:27:13 +00:00
tison
af7107565a refactor: use humantime for user-facing options (#2724) 
* refactor: use humantime for user-facing options

Signed-off-by: tison <wander4096@gmail.com>

* unify quote style

Signed-off-by: tison <wander4096@gmail.com>

* taplo fmt

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2023-11-09 13:29:47 +00:00
Lei, HUANG
b53537e69b fix: unify all sst_write_buffer_size usage (#2712) 
* fix: unify all sst_write_buffer_size usage

* fix: some CR comments

* fix: logs
 2023-11-09 11:22:47 +00:00
tison
b9a7c2db7e feat: make PhiAccrualFailureDetector configurable (#2709) 
* feat: make PhiAccrualFailureDetector configurable

Signed-off-by: tison <wander4096@gmail.com>

* propagate

Signed-off-by: tison <wander4096@gmail.com>

* f32 does not implement Eq

Signed-off-by: tison <wander4096@gmail.com>

* add to test

Signed-off-by: tison <wander4096@gmail.com>

* fmt clippy

Signed-off-by: tison <wander4096@gmail.com>

* Update src/meta-srv/src/failure_detector.rs

* address comments

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2023-11-09 06:26:33 +00:00
Yingwen
16a3257ada fix: add serde default to MitoConfig (#2650) 
* fix: add serde default to MitoConfig

* docs: update config toml
 2023-10-24 11:24:47 +00:00
ZhangJian He
d9e7b898a3 feat: add walconfig dir back (#2606) 
Signed-off-by: ZhangJian He <shoothzj@gmail.com>
 2023-10-16 11:26:06 +00:00
Yun Chen
f859932745 fix: convert to ReadableSize & Durations (#2594) 
* fix: convert to ReadableSize & Durations

* fix: change more grpc sender/recv message size to ReadableSize

fix: format

fix: cargo fmt

fix: change cmd test to use durations

fix: revert metaclient change

fix: convert default fields in meta client options

fix: human serde meta client durations

* fix: remove milisecond postfix in heartbeat option

* fix: humantime serde on heartbeat

* fix: update config example

* fix: update integration test config

* fix: address pr comments

* fix: fix pr comment on default annotation
 2023-10-13 03:28:29 +00:00
Ruihang Xia
62bcb45787 feat!: change config name from kv_store to metadata_store (#2585) 
featchange config name from kv_store to metadata_store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-12 06:55:09 +00:00
dennis zhuang
0a9972aa9a fix: cache capacity unit in sample config (#2575) 2023-10-11 11:02:39 +00:00
dennis zhuang
ff15bc41d6 feat: improve object storage cache (#2522) 
* feat: refactor object storage cache with moka

* chore: minor fixes

* fix: concurrent issues and invalidate cache after write/delete

* chore: minor changes

* fix: cargo lock

* refactor: rename

* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib

* fix: typo

* chore: style

* fix: toml format

* chore: toml

* fix: toml format

* Update src/object-store/src/layers/lru_cache/read_cache.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: update src/object-store/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: refactor and apply suggestions

* fix: typo

* feat: adds back allow list for caching

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* refactor: wrap inner Accessor with Arc

* chore: remove run_pending_task in read and write path

* chore: the arc is unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-10-08 03:27:49 +00:00
Weny Xu
9282e59a3b fix: re-create heartbeat stream ASAP (#2499) 
* chore: set default connect_timeout_millis to 1000

* fix: re-create heartbeat stream ASAP

* chore: apply suggestions
 2023-09-27 04:00:16 +00:00
Weny Xu
230a3026ad fix: dn doesn't have chance to send a heartbeat to the new leader (#2471) 
* refactor: set meta leader lease secs to 3s

* fix: correct default heartbeat interval

* refactor: ask meta leader in parallel

* feat: configure heartbeat client timeout to 500ms

* fix: trigger to send heartbeat immediately after fail

* fix: fix clippy
 2023-09-26 05:05:38 +00:00
JeremyHi
7310ec0bb3 chore: refactor options (#2476) 2023-09-24 02:12:33 +00:00
Weny Xu
43476e1ff9 refactor: rename coordination to require_lease_before_startup (#2431) 2023-09-18 11:07:42 +00:00
Weny Xu
2c3ff90dbc feat: start services after first heartbeat response processed (#2424) 
* feat: start services after first heartbeat response processed

* refactor: watch changes in RegionAliveKeeper

* feat: add coordination to DatanodeOptions

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: enable coordination in sqlness
 2023-09-18 08:49:26 +00:00
JeremyHi
e7e254cd11 feat: all distributed time together (#2423) 2023-09-17 15:18:52 +00:00
JeremyHi
98a40bae95 feat!: unify naming with options (#2416) 2023-09-17 07:24:57 +00:00
Weny Xu
f386329e29 refactor: introduce DdlTaskExecutor and refactor statement executor (#2341) 
* feat: add kv store option

* refactor: refactor statement executor

* refactor: refactor standalone table creator

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor

* fix: fix RegionDistribution

* feat: build standalone

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1 feat: start datanode with config (#2312) 
* remove memory-catalog and procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* derive serde for MitoConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* start datanode with configs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dir in WalConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add rename field attr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add stupid duplicated mito config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove wrong import

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wired compile error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
dennis zhuang
a3d5931fca feat: unify all protocol options (#2316) 
* feat: unify all protocol options

* feat: adds enable to example configs

* chore: style

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
Weny Xu
c56f5e39cd refactor: set default metasrv procedure retry times to 12 (#2242) 2023-08-26 07:41:15 +00:00
Ning Sun
88247e4284 fix!: resolve residual issues with removing prometheus port (#2227) 
* fix: resolve residual issues when removing prometheus port

* fix: remove prometheus from sample config as well
 2023-08-23 01:49:11 +00:00
dennis zhuang
18fa0e01ed feat: remove checkpoint_on_startup (#2228) 
feat: update flushed manifest version when it is larger
 2023-08-22 13:09:34 +00:00
dennis zhuang
6f40128058 feat!: enable telemetry by default (#2137) 
* feat: remove greptimedb-telemetry feature

* feat: adds enable_telemetry option to metasrv and datanode

* refactor: move data_home from file config to storage config

* feat: store the installation uuid into datanode and metasrv working home

* fix: cargo toml fmt

* test: ignore region failver test when using local fle storage

* test: ignore telemetry reporter in test mode

* feat: print warning log when enabling telemetry

* chore: the telemetry doc link

* chore: remove enable_telemetry from datanode example config file

* refactor: rename GREPTIMEDB_TELEMETRY_CLIENT_REQUEST_TIMEOUT

* chore: rename print_warn_log to print_anonymous_usage_data_disclaimer
 2023-08-11 14:50:40 +00:00
Sunray Ley
090b7e61ca feat: make the gRPC channel between Frontend and Datanode configurable (#2044) 
* feat: expose frontend datanode_client_options

* chore: add configuration options to the configuration file

* refactor(frontend): extract DatanodeOptions to service_config

* refactor(frontend): extract DatanodeOptions to service_config

* style: remove unnecessary suffix in variable name

Co-authored-by: Yingwen <realevenyag@gmail.com>

* feat: use humantime_serde for readable duration

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-01 10:49:41 +00:00
Kree0
8f71ac2172 refactor: move heartbeat configuration into an independent section (#1976) 
refactor: move heartbeat configuration into an independent section in config file

* refactor: move heartbeat configuration into an independent section in config file

* feat: add HeartbeatOptions struct

* test: modify corresponding test case

* chore: modify corresponding example file
 2023-07-17 11:29:02 +08:00
Weny Xu
d9751268aa feat: expose metasrv datanode_client_options (#1965) 
* feat: expose meta datanode_client_options

* chore: apply suggestions from CR
 2023-07-15 14:26:24 +08:00
Weny Xu
735c6390ca feat: implement alter table procedure (#1878) 
* feat: implement alter table procedure

* fix: fix uncaught error

* refactor: move fetch_table/s to table_routes.rs

* refactor: refactor error handling

* chore: apply suggestions from CR

* feat: switch to using alter table procedure

* feat: add table_version

* chore: apply suggestions from CR

* feat: introduce ddl_channel_manager

* chore: update greptime-proto
 2023-07-13 10:41:46 +08:00
Eugene Tolbakov
674bfd85c7 chore(prom)!: rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus (#1931) 
* chore(prom): rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus

* chore: apply clippy suggestions

* chore: adjust format according to rustfmt
 2023-07-12 14:47:09 +08:00
Weny Xu
264c5ea720 feat: meta procedure options (#1937) 
* feat: meta procedure options

* chore: tune meta procedure options in tests

* Update src/common/procedure/Cargo.toml

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-07-12 02:22:08 +00:00
Eugene Tolbakov
ccee60f37d feat(http_body_limit): add initial support for DefaultBodyLimit (#1860) 
* feat(http_body_limit): add initial support for DefaultBodyLimit

* fix: address CR suggestions

* fix: adjust the const for default http body limit

* fix: adjust the toml_str for the test

* fix: address CR suggestions

* fix: body_limit units in example config toml files

* fix: address clippy suggestions
 2023-07-04 20:56:56 +08:00
Cao Zhengjia
e54415e723 feat: Make heartbeat intervals configurable in Frontend and Datanode (#1864) 
* update frontend options and config

* fix format
 2023-07-03 12:08:47 +08:00
Lei, HUANG
a314993ab4 chore: change logstore default config (#1809) 2023-06-21 07:34:24 +00:00
dennis zhuang
f08f726bec test: s3 manifest (#1755) 
* feat: change default manifest options

* test: s3 manifest

* feat: revert checkpoint_margin to 10

* Update src/object-store/src/test_util.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-06-09 10:28:41 +00:00
dennis zhuang
ae8203fafa fix: prepare statement doesn't support insert clause (#1680) 
* fix: insert clause doesn't support prepare statement

* fix: manifeste dir

* fix: format

* fix: temp path
 2023-05-31 20:14:58 +08:00
Yingwen
9d179802b8 feat: Add a global TTL option for all tables (#1679) 
* feat: Add a global TTL option for all tables

* docs: update config examples

* chore: print start command and options when standalone/frontend starts
 2023-05-31 15:36:25 +08:00
dennis zhuang
7c55783e53 feat!: reorganize the storage layout (#1609) 
* feat: adds data_home to DataOptions

* refactor: split out object store stuffs from datanode instance

* feat: move data_home into FileConfig

* refactor: object storage layers

* feat: adds datanode path to procedure paths

* feat: temp commit

* refactor: clean code

* fix: forgot files

* fix: forgot files

* Update src/common/test-util/src/ports.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: compile error

* chore: cr comments

* fix: dependencies order in cargo

* fix: data path in test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-23 13:58:26 +08:00
Yingwen
6dbceb1ad5 feat: Trigger flush based on global write buffer size (#1585) 
* feat(storage): Add AllocTracker

* feat(storage): flush request wip

* feat(storage): support global write buffer size

* fix(storage): Test and fix size based strategy

* test(storage): Test AllocTracker

* test(storage): Test pick_by_write_buffer_full

* docs: Add flush config example

* test(storage): Test schedule_engine_flush

* feat(storage): Add metrics for write buffer size

* chore(flush): Add log when triggering flush by global buffer

* chore(storage): track allocation in update_stats
 2023-05-22 19:00:30 +08:00
Yingwen
8d54d40b21 feat: Add FlushPicker to flush regions periodically (#1559) 
* feat: Add FlushPicker

* feat(storage): Add close to StorageEngine

* style(storage): fix clippy

* feat(storage): Close regions in StorageEngine::close

* chore(storage): Clear requests on scheduler stop

* test(storage): Test flush picker

* feat(storage): Add metrics for auto flush

* feat(storage): Add flush reason and record it in metrics

* feat: Expose flush config

docs(config): Update config example

* refactor(storage): Run auto flush task in FlushScheduler

* refactor(storage): Add FlushItem trait to make FlushPicker easy to test
 2023-05-15 17:29:28 +08:00
Yingwen
0b0b5a10da feat: Remove store from procedure config (#1489) 
* feat(procedure): Add key prefix

* feat: Remove store config from ProcedureConfig

* refactor(procedure): Address review comments

Add proc_path! macro and rename KEY_PREFIX to PROC_PATH

* docs: Update procedure config examples
 2023-04-28 22:12:57 +08:00
Vanish
9f0efc748d feat: make log level and destination configurable from config files (#1444) 
* feat: implement load_options.

* refactor: build by ConfigOptions.

* refactor: init_global_logging by LoggingOptions.

* chore: make clippy happy.

* refactor: use TopLevelOptions push top level options to subcommand.

* test: test TopLevelOptions.

* refactor: push Options in Box.

* refactor: push Options in Box.

* refactor: use let-else and Options.
 2023-04-27 15:30:04 +08:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
dennis zhuang
bf8c717022 feat: try to do manifest checkpoint on opening region (#1321) 2023-04-04 21:36:54 +08:00
dennis zhuang
563adbabe9 feat!: improve region manifest service (#1268) 
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-31 10:42:00 +08:00
Xieqijun
cd8be77968 feat(procedure): Max retry time (#1095) 
* feat: procedure config

* fix: modify config

* feat: add retry logic

* feat: add error

* feat: add it

* feat: add it

* feat: add it

* feat: rm retry from runner

* feat: use backon

* feat: add retry_interval

* feat: add retry_interval

* fix: conflict

* fix: cr

* feat: add retry error and id

* feat: rename

* refactor: execute

* feat: use config dir

* fix: cr

* fix: cr

* fix: fmt

* fix: fmt

* fix: pr

* fix: it

* fix: rm unless cmd params

* feat: add toml

* fix: ut

* feat: add rolling back

* fix: cr

* fix: cr

* fix: cr

* fix: ci

* fix: ci

* fix: ci

* chore: Apply suggestions from code review

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-03-15 08:28:08 +00:00