rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-07 13:52:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,806 Commits 104 Branches 226 Tags
e463942a5bfef7a7b7f052d3bbb8016719d906eb
Commit Graph

119 Commits

Author SHA1 Message Date
Zhenchi
421e38c481 feat: allow purging a given puffin file in staging area (#5558) 
* feat: purge a given puffin file in staging area

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish log

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* ttl set to 2d

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: expose staging_ttl to index config

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* use `invalidate_entries_if` instead of maintaining map

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* run_pending_tasks after purging

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
 2025-02-19 08:58:30 +00:00
zyy17
954310f917 feat: implement Jaeger query APIs (#5452) 
* feat: implement jaeger query api

* test: add some unit tests

* test: add integration tests for jaeger query APIs

* refactor: parse tags from url parameters

* refactor: support to query traces by tags

* refactor: add limit parameter

* refactor: add jaeger query api metrics

* chore: add some comment docs and default limit value

* test: add more unit tests

* docs: add jaeger options in config docs

* refactor: code review

* wip

* refactor: use datafusion's dataframe APIs to query traces

* refactor: code review

* chore: format test cases

* refactor: add check_schema()

* chore: fix clippy errors and rename function name

* refactor: throw error when covert start_time and duration error

* chore: modify incorrect request type name

* chore: remove unecessary serde rename

* refactor: add some important comments

* refactor: add SPAN_KIND_PREFIX

* refactor: code review
 2025-02-12 23:36:38 +00:00
Ruihang Xia
e22aa819be feat: support server-side keep-alive for mysql and pg protocols (#5496) 
* feat: support server-side keep-alive for mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config.md

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config to use humantime for keep-alive configuration

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: Update socket2 dependency

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-11 19:22:10 +00:00
Weny Xu
c80d2a3222 fix: introduce gc task for metadata store (#5461) 
* fix: introduce gc task for metadata kvbackend

* refactor: refine KvbackendConfig

* chore: apply suggestions from CR
 2025-02-06 12:12:43 +00:00
jeremyhi
fac22575aa refactor!: unify the option names across all components (#5457) 
* refactor: rename grpc options

* refactor: make the arg clearly

* chore: comments on server_addr

* chore: fix test

* chore: remove the store_addr alias

* refactor: cli option rpc_server_addr

* chore: keep store-addr alias

* chore: by comment
 2025-02-06 06:37:14 +00:00
Ning Sun
698b28c636 feat: provide options to disable or customize http corss-origin settings (#5450) 
* feat: add cors headers for http server

* test: add cors test

* test: add preflight test

* feat: allow customize http cross-origin settings

* chore: typo fix

* test: update tests

* test: fix test for config

* refactor: address review comments
 2025-01-26 03:55:34 +00:00
Weny Xu
4323c20d18 feat(metric-engine): introduce experimental_sparse_primary_key_encoding to MetricEngineConfig (#5373) 
* feat: introduce `experimental_sparse_primary_key_encoding` to `MetricEngineConfig`

* fix: unit test
 2025-01-20 03:49:39 +00:00
Weny Xu
ccd2b06b7a fix!: correct index cache config (#5381) 
* fix: correct index cache config

* chore: update config.md
 2025-01-16 09:46:02 +00:00
discord9
66f63ae981 feat: more than one flow workers (#5315) 
* feat: more workers

* feat: use round robin

* refactor: per review

* refactor: per bot review

* chore: per review

* docs: example

* docs: update config.md

* docs: update

* chore: per review

* refactor: set workers to cpu/2.max(1)

* fix: flow config in standalone mode

* test: fix config test

* docs: update docs&opt name

* chore: update config.md

* refactor: per review, sanitize at top

* chore: per review

* chore: config.md
 2025-01-14 12:28:18 +00:00
Yingwen
69d9a2845f docs: update standalone example (#5290) 2025-01-04 11:30:25 +00:00
Kould
1067357b72 chore(config)!: refactor configs of write cache (#5259) 
* chore: refactor configs of write cache

* chore: write_cache_size `10GiB` -> `5GiB`
 2025-01-04 07:14:38 +00:00
Yohan Wal
bcb0f14227 refactor: adjust index cache page size (#5267) 
* refactor: adjust index cache page size

* fix: wrong docs

* Update config/datanode.example.toml

* Update config/config.md

* Update config/config.md

* chore: adjust to 64KiB

* Apply suggestions from code review
 2025-01-03 03:26:17 +00:00
Zhenchi
f4b2d393be feat(config): add bloom filter config (#5237) 
* feat(bloom-filter): integrate indexer with mito2

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(config) add bloom filter config

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* merge

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove cache config

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-12-26 04:38:45 +00:00
zyy17
5e88c80394 feat: introduce the Limiter in frontend to limit the requests by in-flight write bytes size. (#5231) 
feat: introduct Limiter to limit in-flight write bytes size in frontend
 2024-12-25 09:11:30 +00:00
Yohan Wal
4b4c6dbb66 refactor: cache inverted index with fixed-size page (#5114) 
* feat: cache inverted index by page instead of file

* fix: add unit test and fix bugs

* chore: typo

* chore: ci

* fix: math

* chore: apply review comments

* chore: renames

* test: add unit test for index key calculation

* refactor: use ReadableSize

* feat: add config for inverted index page size

* chore: update config file

* refactor: handle multiple range read and fix some related bugs

* fix: add config

* test: turn to a fs reader to match behaviors of object store
 2024-12-13 07:34:24 +00:00
dennis zhuang
2c4ac76754 feat: adjust WAL purge default configurations (#5107) 
* feat: adjust WAL purge default configurations

* fix: config

* feat: change raft engine file_size default to 128Mib
 2024-12-11 08:08:05 +00:00
dennis zhuang
03a28320d6 feat!: enable read cache and write cache when using remote object stores (#5093) 
* feat: enable read cache and write cache when using remote object stores

* feat: make read cache be aware of remote store names

* chore: docs

* chore: apply review suggestions

* chore: trim write cache path

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-12-10 04:03:44 +00:00
Yingwen
2fcb95f50a fix!: fix regression caused by unbalanced partitions and splitting ranges (#5090) 
* feat: assign partition ranges by rows

* feat: balance partition rows

* feat: get uppoer bound for part nums

* feat: only split in non-compaction seq scan

* fix: parallel scan on multiple sources

* fix: can split check

* feat: scanner prepare by request

* feat: remove scan_parallelism

* docs: upate docs

* chore: update comment

* style: fix clippy

* feat: skip merge and dedup if there is only one source

* chore: Revert "feat: skip merge and dedup if there is only one source"

Since memtable won't do dedup jobs

This reverts commit 2fc7a54b11.

* test: avoid compaction in sqlness window sort test

* chore: do not create semaphore if num partitions is enough

* chore: more assertions

* chore: fix typo

* fix: compaction flag not set

* chore: address review comments
 2024-12-09 12:50:57 +00:00
LFC
57f31d14c8 refactor: expose configs for http clients used in object store (#5041) 2024-11-25 03:49:54 +00:00
dennis zhuang
edc49623de chore: update default cache size to 1Gib (#4923) 
* chore: update default cache size to 1Gib for object storage read/write cache

* feat: update docs

* fix: test
 2024-11-04 03:53:17 +00:00
Ruihang Xia
9712295177 fix(config): update tracing section headers in example TOML files (#4898) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-10-30 08:31:31 +00:00
Yingwen
1008af5324 feat!: Divide flush and compaction job pool (#4871) 
* feat: divide flush/compact job pool

* feat!: divide bg jobs config

* docs: update config examples

* test: fix tests
 2024-10-25 23:36:16 +00:00
Weny Xu
ca6e02980e fix: overwrite entry_id if entry id is less than start_offset (#4842) 
* fix: overwrite entry_id if entry id is less than start_offset

* feat: add `overwrite_entry_start_id` to options

* chore: update config.md
 2024-10-18 06:31:02 +00:00
Kaifeng Zheng
a283e13da7 feat: set max log files to 720 by default, info log only (#4787) 
* feat: set max log files to 720 by default, info log only

* expose max_log_files in tomls

* include dir info when panicing, limit max_log_files of err_log to 30, and that of slow_queries to opt.max_log_files

* fix clippy

* update config.md

* update expected config str

* limit err_log max files size to `max_log_files` too, include err info when panicing, put `max_l_f` in right position

* fix typos

* chore: config

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-10-04 18:05:40 +00:00
zyy17
e39a9e6feb feat: add StatementStatistics for slow query logging implementation (#4719) 
* feat: log slow query

* feat: log slow query for sql

* refactor: add slow query logging options

* ci: fix errors

* feat: add StatementStatistics

* chore: revert modification of servers crate

* docs: update config docs

* fix: clippy errors
 2024-09-30 03:26:50 +00:00
Lanqing Yang
cc4106cbd2 feat: protect datanode with concurrency limit. (#4699) 
Adding parallelism in region server to protect datanode from query overload.
 2024-09-27 06:12:57 +00:00
zyy17
aa03d3b11c docs: use docs comment prefix and bump toml2docs version (#4711) 2024-09-11 07:49:23 +00:00
Lei, HUANG
208afe402b feat(wal): increase recovery parallelism (#4689) 
* Refactor RaftEngineLogStore to use references for config

 - Updated `RaftEngineLogStore::try_new` to accept a reference to `RaftEngineConfig` instead of taking ownership.
 - Replaced direct usage of `config` with individual fields (`sync_write`, `sync_period`, `read_batch_size`).
 - Adjusted test cases to pass references to `RaftEngineConfig`.

* Add parallelism configuration for WAL recovery

 - Introduced `recovery_parallelism` setting in `datanode.example.toml` and `standalone.example.toml` for configuring parallelism during WAL recovery.
 - Updated `Cargo.lock` and `Cargo.toml` to include `num_cpus` dependency.
 - Modified `RaftEngineConfig` to include `recovery_parallelism` with a default value set to the number of CP

* feat/wal-recovery-parallelism:
 Add `wal.recovery_parallelism` configuration option

 - Introduced `wal.recovery_parallelism` to config.md for specifying parallelism during WAL recovery.
 - Updated `RaftEngineLogStore` to include `recovery_threads` from the new configuration.

* fix: ut
 2024-09-09 04:25:24 +00:00
LFC
d43e31c7ed feat: schedule compaction when adding sst files by editing region (#4648) 
* feat: schedule compaction when adding sst files by editing region

* add minimum time interval for two successive compactions

* resolve PR comments
 2024-09-04 10:10:07 +00:00
zyy17
8453df1392 refactor: make init_global_logging() clean and add log_format (#4657) 
refactor: refine the code logic of init_global_logging and add json output format
 2024-09-04 03:04:51 +00:00
dennis zhuang
a612b67470 feat: supports name in object storage config (#4630) 
* feat: supports name in object storage config

* fix: integration test

* fix: integration test

* fix: update sample config

* fix: config api test
 2024-09-03 07:02:55 +00:00
Weny Xu
306bd25c64 fix: expose missing options for initializing regions (#4660) 
* fix: expose `init_regions_in_background` and `init_regions_parallelism` opts

* fix: ci
 2024-09-02 03:11:18 +00:00
dennis zhuang
841e66c810 fix: config api and export metrics default database (#4633) 2024-08-28 14:28:49 +00:00
Weny Xu
3297d5f657 feat: allow skipping topic creation (#4616) 
* feat: introduce `create_topics` opt

* feat: allow skipping topic creation

* chore: refine docs

* chore: apply suggestions from CR
 2024-08-26 08:34:27 +00:00
Yingwen
a12a905578 chore: disable ttl for write cache by default (#4595) 
* chore: remove default write cache ttl

* docs: update example config

* chore: fix ci
 2024-08-21 08:38:38 +00:00
liyang
f578155602 feat: add GcsConfig credential field (#4568) 2024-08-16 03:11:20 +00:00
Weny Xu
2e2eacf3b2 feat: add SASL and TLS config for Kafka client (#4536) 
* feat: add SASL and TLS config

* feat: add SASL/PLAIN and TLS config for Kafka client

* chore: use `ring`

* feat: support SASL SCRAM-SHA-256 and SCRAM-SHA-512

* fix: correct unit test

* test: add integration test

* chore: apply suggestions from CR

* refactor: introduce `KafkaConnectionConfig`

* chore: refine toml examples

* docs: add missing fields

* chore: refine examples

* feat: allow no server ca cert

* chore: refine examples

* chore: fix clippy

* feat: load system ca certs

* chore: fmt toml

* chore: unpin version

* Update src/common/wal/src/error.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-08-12 12:27:11 +00:00
Weny Xu
a25d9f736f chore: set default otlp_endpoint (#4508) 
* chore: set default `otlp_endpoint`

* fix: fix ci
 2024-08-06 06:48:14 +00:00
Ruihang Xia
7daf24c47f feat: remove dedicated runtime for grpc, mysql and pg protocols (#4436) 
* feat: remove dedicated runtime for grpc, mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove other runtimes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* spawn compact task into compact_runtime

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refine naming

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/servers/tests/mysql/mysql_server_test.rs

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* turnoff fuzz test matrix fail fast option

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update rt config for ci tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-30 06:17:58 +00:00
Lei, HUANG
bdd3d2d9ce chore: add dynamic cache size adjustment for InvertedIndexConfig (#4433) 
* Add dynamic cache size adjustment for InvertedIndexConfig

* Increase cache sizes in integration tests for HTTP

 - Updated `metadata_cache_size` from 32MiB to 64MiB

* Remove cache size settings from config and update drop_lines_with_inconsistent_results function to handle them

* Add cache size configurations for inverted index metadata and content

 - Introduced `metadata_cache_size` with a default of 64MiB.
 - Introduced `content_cache_size` with a default of 128MiB.

* chore/index-content-cache-default-size: Add cache size configuration options for Mito engine's inverted index
 2024-07-26 03:36:20 +00:00
Weny Xu
7ad248d6f6 fix(config): enable file engine by default (#4345) 
* fix: enable file engine by default

* fix: fix tests
 2024-07-11 17:35:18 +00:00
Zhenchi
a710676d06 feat(fulltext_index): integrate full-text indexer with sst writer (#4302) 
* feat(fulltext_index): integrate full-text indexer with sst writer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: delay building puffin writer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: indexer test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add abort on empty indexer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: indicates default mode

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: introduce "auto" and "unlimited" as mem threshold

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* doc: comment about push empty string

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-07 04:10:19 +00:00
Zhenchi
65c9fbbd2f feat(fulltext_index): integrate puffin manager with inverted index applier (#4266) 
* feat(fulltext_index): integrate puffin manager with inverted index applier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: get rid of unexpected not found from write cache

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: move create_dir_all to BoundedStager::new

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: update config.md

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* config: unify directories

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: silent remove

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: config docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: auxiliary -> aux

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-04 06:18:58 +00:00
Jeremyhi
df0fff2f2c feat(servers): make http timeout and body limit optional (#4217) 
* feat(servers): make http timeout and body limit optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: make config-docs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-26 06:14:14 +00:00
Weny Xu
ea2d067cf1 feat: implement the OrderedBatchProducer (#4134) 
* feat: implement the `OrderedBatchProducer`

* test: add test of cancel safety

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: simplify the `BackgroundProducerWorker`

* feat: implement the OrderedBatchProducer v2

* refactor: switch to `OrderedBatchProducer`

* chore: rename to `MAX_FLUSH_QUEUE_SIZE`

* refactor: switch to `OrderedBatchProducerV2`

* refactor: remove `OrderedBatchProducerV1`

* test: add tests

* refactor: make config configurable

* refactor: minor refactor

* chore: remove unused code

* chore: remove `benchmarks` crate

* chore: update config doc

* chore: remove unused comment

* refactor: refactor client registry

* refactor: rename `max_batch_size` to `max_batch_bytes`

* chore: use constant value

* chore: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: remove the `CHANNEL_SIZE`

* chore: apply suggestions from CR

* fix: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-06-18 07:20:01 +00:00
Ning Sun
70d113a355 feat: update default size of bgworkers, add hbworkers (#4129) 
* feat: update default size of bgworkers, add hbworkers

* feat: update frontend heartbeat as well

* chore: update sample config files and default settings

* chore: update config docs

* Revert "chore: update config docs"

This reverts commit 8107f4c120.

* Revert "chore: update sample config files and default settings"

This reverts commit f5ae701c8d.

* feat: use default heartbeat runtime size

* chore: update config docs
 2024-06-18 06:18:37 +00:00
LFC
c0aed1d267 feat: set global runtime size by config file (#4063) 
* set global runtime size

* fix: resolve PR comments

* fix: log the whole option

* fix ci

* debug ci

* debug ci

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-06-04 10:03:33 +00:00
taobo
a3a2c8d063 feat: Add TLS support for gRPC service (#3957) 
* feat: Add tls support for grpc service

* feat: add integration test

* fix: integration test

* fix: revert by suggestion

* fix: typos

* fix: optimize code

* fix: optimize code

* docs: update configs
 2024-05-23 19:00:16 +00:00
shuiyisong
9e1af79637 chore: add ttl to write_cache (#4010) 
* chore: add ttl to write_cache

* chore: update test & add example config

* chore: fix typo

* chore: fix typo

* chore: fix typo
 2024-05-22 06:50:12 +00:00
zyy17
60eb5de3f1 refactor: add tracing options in xOptions (#3919) 
* refactor: add tracing options in {DatanodeOptions, FrontendOptions, MetasrvOptions, StandaloneOptions}

* ci: fix integration error

* refactor: minor modification for initialization of tracing options
 2024-05-13 05:16:50 +00:00