rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-06 21:32:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,991 Commits 105 Branches 226 Tags
69659211f6066c1c021f6caeca4b16bda0fec785
Commit Graph

258 Commits

Author SHA1 Message Date
Lanqing Yang
cc4106cbd2 feat: protect datanode with concurrency limit. (#4699) 
Adding parallelism in region server to protect datanode from query overload.
 2024-09-27 06:12:57 +00:00
zyy17
aa03d3b11c docs: use docs comment prefix and bump toml2docs version (#4711) 2024-09-11 07:49:23 +00:00
Lei, HUANG
208afe402b feat(wal): increase recovery parallelism (#4689) 
* Refactor RaftEngineLogStore to use references for config

 - Updated `RaftEngineLogStore::try_new` to accept a reference to `RaftEngineConfig` instead of taking ownership.
 - Replaced direct usage of `config` with individual fields (`sync_write`, `sync_period`, `read_batch_size`).
 - Adjusted test cases to pass references to `RaftEngineConfig`.

* Add parallelism configuration for WAL recovery

 - Introduced `recovery_parallelism` setting in `datanode.example.toml` and `standalone.example.toml` for configuring parallelism during WAL recovery.
 - Updated `Cargo.lock` and `Cargo.toml` to include `num_cpus` dependency.
 - Modified `RaftEngineConfig` to include `recovery_parallelism` with a default value set to the number of CP

* feat/wal-recovery-parallelism:
 Add `wal.recovery_parallelism` configuration option

 - Introduced `wal.recovery_parallelism` to config.md for specifying parallelism during WAL recovery.
 - Updated `RaftEngineLogStore` to include `recovery_threads` from the new configuration.

* fix: ut
 2024-09-09 04:25:24 +00:00
LFC
d43e31c7ed feat: schedule compaction when adding sst files by editing region (#4648) 
* feat: schedule compaction when adding sst files by editing region

* add minimum time interval for two successive compactions

* resolve PR comments
 2024-09-04 10:10:07 +00:00
zyy17
8453df1392 refactor: make init_global_logging() clean and add log_format (#4657) 
refactor: refine the code logic of init_global_logging and add json output format
 2024-09-04 03:04:51 +00:00
dennis zhuang
a612b67470 feat: supports name in object storage config (#4630) 
* feat: supports name in object storage config

* fix: integration test

* fix: integration test

* fix: update sample config

* fix: config api test
 2024-09-03 07:02:55 +00:00
Weny Xu
306bd25c64 fix: expose missing options for initializing regions (#4660) 
* fix: expose `init_regions_in_background` and `init_regions_parallelism` opts

* fix: ci
 2024-09-02 03:11:18 +00:00
dennis zhuang
841e66c810 fix: config api and export metrics default database (#4633) 2024-08-28 14:28:49 +00:00
Weny Xu
3297d5f657 feat: allow skipping topic creation (#4616) 
* feat: introduce `create_topics` opt

* feat: allow skipping topic creation

* chore: refine docs

* chore: apply suggestions from CR
 2024-08-26 08:34:27 +00:00
Lanqing Yang
ff04109ee6 docs: add example configs introduced by pg_kvbackend (#4573) 
chore: add example configs that introduced after pg_kvbackend
 2024-08-22 01:52:02 +00:00
Yingwen
a12a905578 chore: disable ttl for write cache by default (#4595) 
* chore: remove default write cache ttl

* docs: update example config

* chore: fix ci
 2024-08-21 08:38:38 +00:00
Weny Xu
76dc906574 feat(log_store): introduce the CollectionTask (#4530) 
* feat: introduce the `CollectionTask`

* feat: add config of index collector

* chore: remove unused code

* feat: truncate indexes

* chore: apply suggestions from CR

* chore: update config examples

* refactor: retrieve latest offset while dumping indexes

* chore: print warn
 2024-08-19 03:48:35 +00:00
liyang
f578155602 feat: add GcsConfig credential field (#4568) 2024-08-16 03:11:20 +00:00
Weny Xu
2e2eacf3b2 feat: add SASL and TLS config for Kafka client (#4536) 
* feat: add SASL and TLS config

* feat: add SASL/PLAIN and TLS config for Kafka client

* chore: use `ring`

* feat: support SASL SCRAM-SHA-256 and SCRAM-SHA-512

* fix: correct unit test

* test: add integration test

* chore: apply suggestions from CR

* refactor: introduce `KafkaConnectionConfig`

* chore: refine toml examples

* docs: add missing fields

* chore: refine examples

* feat: allow no server ca cert

* chore: refine examples

* chore: fix clippy

* feat: load system ca certs

* chore: fmt toml

* chore: unpin version

* Update src/common/wal/src/error.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-08-12 12:27:11 +00:00
Ning Sun
9532ffb954 fix: configuration example for selector (#4532) 
* fix: configuration example for selector

* docs: update config docs

* test: update unit tests for configuration in meta
 2024-08-09 09:51:05 +00:00
Weny Xu
a25d9f736f chore: set default otlp_endpoint (#4508) 
* chore: set default `otlp_endpoint`

* fix: fix ci
 2024-08-06 06:48:14 +00:00
Ruihang Xia
7daf24c47f feat: remove dedicated runtime for grpc, mysql and pg protocols (#4436) 
* feat: remove dedicated runtime for grpc, mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove other runtimes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* spawn compact task into compact_runtime

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refine naming

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/servers/tests/mysql/mysql_server_test.rs

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* turnoff fuzz test matrix fail fast option

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update rt config for ci tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-30 06:17:58 +00:00
zyy17
1a38f36d2d refactor!: Remove Mode from FrontendOptions (#4401) 
refactor: remove `Mode` from `FrontendOptions`

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2024-07-29 06:57:01 +00:00
Lei, HUANG
bdd3d2d9ce chore: add dynamic cache size adjustment for InvertedIndexConfig (#4433) 
* Add dynamic cache size adjustment for InvertedIndexConfig

* Increase cache sizes in integration tests for HTTP

 - Updated `metadata_cache_size` from 32MiB to 64MiB

* Remove cache size settings from config and update drop_lines_with_inconsistent_results function to handle them

* Add cache size configurations for inverted index metadata and content

 - Introduced `metadata_cache_size` with a default of 64MiB.
 - Introduced `content_cache_size` with a default of 128MiB.

* chore/index-content-cache-default-size: Add cache size configuration options for Mito engine's inverted index
 2024-07-26 03:36:20 +00:00
Weny Xu
89b86c87a2 chore: add docs for config file (#4432) 2024-07-25 08:11:10 +00:00
Weny Xu
05c7d3eb42 docs(config): add enable_region_failover option to configuration (#4355) 
docs(config): Add enable_region_failover option to configuration
 2024-07-12 08:09:25 +00:00
Weny Xu
7ad248d6f6 fix(config): enable file engine by default (#4345) 
* fix: enable file engine by default

* fix: fix tests
 2024-07-11 17:35:18 +00:00
discord9
1ddf19d886 feat: flownode use Inserter to write to database (#4323) 
* feat: use `Inserter` as Frontend

* fix: enable procedure in flownode

* docs: remove `frontend_addr` opts

* chore: rm fe addr in test runner

* refactor: int test also use inserter invoker

* feat: flow shutdown&refactor: remove `Frontendinvoker`

* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`

* refactor: per review

* refactor: remove a layer of  box

* fix: standalone use `node_manager`

* fix: remove a `Arc` cycle
 2024-07-09 10:44:22 +00:00
Zhenchi
a710676d06 feat(fulltext_index): integrate full-text indexer with sst writer (#4302) 
* feat(fulltext_index): integrate full-text indexer with sst writer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: delay building puffin writer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: indexer test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add abort on empty indexer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: indicates default mode

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: introduce "auto" and "unlimited" as mem threshold

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* doc: comment about push empty string

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-07 04:10:19 +00:00
discord9
09fff24ac4 feat: make flow distributed work&tests (#4256) 
feat: flownode frontend client&test

feat: Frontend Client

feat: set frontend invoker for flownode

feat: set frontend invoker for flownode

chore: test script

WIP: test flow distributed

feat: hard coded demo

docs: flownode example toml

feat: add flownode support in runner

docs: comments for node

chore: after rebase

docs: add a todo

tests: move flow tests to common

fix: flownode sqlness dist test

chore: per review

docs: make

fix: make doc
 2024-07-05 14:46:44 +00:00
Zhenchi
65c9fbbd2f feat(fulltext_index): integrate puffin manager with inverted index applier (#4266) 
* feat(fulltext_index): integrate puffin manager with inverted index applier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: get rid of unexpected not found from write cache

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: move create_dir_all to BoundedStager::new

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: update config.md

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: unify directories

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: silent remove

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: config docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: auxiliary -> aux

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-04 06:18:58 +00:00
Weny Xu
6a634f8e5d feat: register & deregister region failure detectors actively (#4223) 
* feat: Use DATANODE_LEASE_SECS from distributed_time_constants for heartbeat pause duration

* feat: introduce `RegionFailureDetectorController` to manage region failure detectors

* feat: add `RegionFailureDetectorController` to `DdlContext`

* feat: add `region_failure_detector_controller` to `Context` in region migration

* feat: register region failure detectors during rollback region migration procedure

* feat: deregister region failure detectors during drop table procedure

* feat: register region failure detectors during create table procedure

* fix: update meta config

* chore: apply suggestions from CR

* chore: avoid cloning

* chore: rename

* chore: reduce the size of the test

* chore: apply suggestions from CR

* chore: move channel initialization into `RegionSupervisor::channel`

* chore: minor refactor

* chore: rename ident
 2024-07-01 05:58:27 +00:00
Jeremyhi
df0fff2f2c feat(servers): make http timeout and body limit optional (#4217) 
* feat(servers): make http timeout and body limit optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: make config-docs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-26 06:14:14 +00:00
dennis zhuang
a779cb36ec fix: wrong frontend registration address (#4199) 
* fix: frontend registration address is wrong, #4186

* fix: license header

* chore: adds hostname to frontend grpc

* fix: forgot run make config-docs

* chore: warn when using bind_addr

* fix: flow node heartbeat carrying address
 2024-06-26 06:13:07 +00:00
taobo
22d12683b4 refactor!: unify FrontendOptions and DatanodeOptions by using GrpcOptions (#4088) 
* refactor: move GrpcOptions to servers/grpc

* fix: optimize code

* fix: docs

* refactor: move DatanodeOptions.rpc_hostname to grpc.hostname

* fix: merge main

* refactor code impl

test: add test_depreacted_cli_options unit test

* Update src/servers/src/grpc.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-18 22:45:38 +00:00
Weny Xu
ea2d067cf1 feat: implement the OrderedBatchProducer (#4134) 
* feat: implement the `OrderedBatchProducer`

* test: add test of cancel safety

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: simplify the `BackgroundProducerWorker`

* feat: implement the OrderedBatchProducer v2

* refactor: switch to `OrderedBatchProducer`

* chore: rename to `MAX_FLUSH_QUEUE_SIZE`

* refactor: switch to `OrderedBatchProducerV2`

* refactor: remove `OrderedBatchProducerV1`

* test: add tests

* refactor: make config configurable

* refactor: minor refactor

* chore: remove unused code

* chore: remove `benchmarks` crate

* chore: update config doc

* chore: remove unused comment

* refactor: refactor client registry

* refactor: rename `max_batch_size` to `max_batch_bytes`

* chore: use constant value

* chore: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: remove the `CHANNEL_SIZE`

* chore: apply suggestions from CR

* fix: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-06-18 07:20:01 +00:00
Ning Sun
70d113a355 feat: update default size of bgworkers, add hbworkers (#4129) 
* feat: update default size of bgworkers, add hbworkers

* feat: update frontend heartbeat as well

* chore: update sample config files and default settings

* chore: update config docs

* Revert "chore: update config docs"

This reverts commit 8107f4c120.

* Revert "chore: update sample config files and default settings"

This reverts commit f5ae701c8d.

* feat: use default heartbeat runtime size

* chore: update config docs
 2024-06-18 06:18:37 +00:00
Weny Xu
e142ca40d7 feat: invoke handle_batch_open_requests (#4107) 
* feat: open all regions via invoking `handle_batch_open_requests`

* tests: add sqlness tests

* refactor: avoid cloning

* chore: apply suggestions from CR

* chore: update config.md

* chore: apply suggestions from CR
 2024-06-07 09:07:45 +00:00
LFC
c0aed1d267 feat: set global runtime size by config file (#4063) 
* set global runtime size

* fix: resolve PR comments

* fix: log the whole option

* fix ci

* debug ci

* debug ci

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-06-04 10:03:33 +00:00
taobo
a3a2c8d063 feat: Add TLS support for gRPC service (#3957) 
* feat: Add tls support for grpc service

* feat: add integration test

* fix: integration test

* fix: revert by suggestion

* fix: typos

* fix: optimize code

* fix: optimize code

* docs: update configs
 2024-05-23 19:00:16 +00:00
shuiyisong
9e1af79637 chore: add ttl to write_cache (#4010) 
* chore: add ttl to write_cache

* chore: update test & add example config

* chore: fix typo

* chore: fix typo

* chore: fix typo
 2024-05-22 06:50:12 +00:00
zyy17
b8354bbb55 docs: add toc for config docs (#3974) 2024-05-18 01:57:49 +00:00
Ruihang Xia
4b030456f6 feat: remove timeout in the channel between frontend and datanode (#3962) 
* style: change builder pattern

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: remove timeout

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update docs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-16 14:12:42 +00:00
zyy17
60eb5de3f1 refactor: add tracing options in xOptions (#3919) 
* refactor: add tracing options in {DatanodeOptions, FrontendOptions, MetasrvOptions, StandaloneOptions}

* ci: fix integration error

* refactor: minor modification for initialization of tracing options
 2024-05-13 05:16:50 +00:00
shuiyisong
6e9e8fad26 refactor!: remove opentsdb tcp server (#3828) 
* refactor: remove opentsdb tcp server

* refactor: remove config and add test

* refactor: update docs and remove unused code
 2024-05-06 06:42:05 +00:00
Yingwen
d2e081c1f9 docs: update memtable config example (#3712) 2024-04-16 07:26:20 +00:00
zyy17
8060c81e1d refactor: use toml2docs to generate config docs (#3704) 
* refactor: use toml2docs to generate config docs

* ci: add docs check in 'check-typos-and-docs'
 2024-04-15 09:08:32 +00:00
Lei, HUANG
183fccbbd6 chore: remove global_ttl config (#3673) 
* chore: remove global_ttl config

* fix: clippy
 2024-04-09 02:00:50 +00:00
Weny Xu
a0f243c128 feat(procedure): enable auto split large value (#3628) 
* chore: add comments

* chore: remove `pub`

* chore: rename to `merge_multiple_values`

* chore: fix typo

* feat(procedure): enable auto split large value

* chore: apply suggestions from CR

* chore: rename to `max_metadata_value_size`

* chore: remove the NoneAsEmptyString

* chore: set default max_metadata_value_size to 1500KiB
 2024-04-02 12:13:59 +00:00
Yingwen
39b69f1e3b refactor!: Renames the new memtable to PartitionTreeMemtable (#3547) 
* refactor: rename mod merge_tree to partition_tree

* refactor: rename merge_tree

* refactor: change merge tree comment

* refactor: rename merge tree struct

* refactor: memtable options
 2024-03-20 06:40:41 +00:00
Lei, HUANG
7183fa198c refactor: make MergeTreeMemtable the default choice (#3430) 
* refactor: make MergeTreeMemtable the default choice

* refactor: reformat

* chore: add doc to config
 2024-03-05 10:00:08 +00:00
WU Jingdi
53f2a5846c feat: support tracing rule sampler (#3405) 
* feat: support tracing rule sampler

* chore: simplify code
 2024-03-05 15:40:02 +08:00
Ning Sun
d4a54a085b feat: add configuration for tls watch option (#3395) 
* feat: add configuration for tls watch option

* test: sleep longer to ensure async task run

* test: update config api integration test

* refactor: rename function
 2024-03-01 03:49:54 +00:00
Zhenchi
c3c80b92c8 feat(index): measure memory usage in global instead of single-column and add metrics (#3383) 
* feat(index): measure memory usage in global instead of single-column and add metrics

* feat: add leading zeros to streamline memory usage

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove println

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-02-28 06:49:24 +00:00
fys
74bfb09195 feat: support cache for batch_get in CachedMetaKvBackend (#3277) 
* feat: support cache for batch_get in CachedMetaKvBackend.

* add doc of CachedMetaKvBackend

* fix: cr

* fix: correct some words

* fix cr
 2024-02-06 03:15:03 +00:00