Ruihang Xia
9712295177
fix(config): update tracing section headers in example TOML files ( #4898 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-10-30 08:31:31 +00:00
Yingwen
1008af5324
feat!: Divide flush and compaction job pool ( #4871 )
...
* feat: divide flush/compact job pool
* feat!: divide bg jobs config
* docs: update config examples
* test: fix tests
2024-10-25 23:36:16 +00:00
Weny Xu
ca6e02980e
fix: overwrite entry_id if entry id is less than start_offset ( #4842 )
...
* fix: overwrite entry_id if entry id is less than start_offset
* feat: add `overwrite_entry_start_id` to options
* chore: update config.md
2024-10-18 06:31:02 +00:00
Kaifeng Zheng
a283e13da7
feat: set max log files to 720 by default, info log only ( #4787 )
...
* feat: set max log files to 720 by default, info log only
* expose max_log_files in tomls
* include dir info when panicing, limit max_log_files of err_log to 30, and that of slow_queries to opt.max_log_files
* fix clippy
* update config.md
* update expected config str
* limit err_log max files size to `max_log_files` too, include err info when panicing, put `max_l_f` in right position
* fix typos
* chore: config
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2024-10-04 18:05:40 +00:00
zyy17
e39a9e6feb
feat: add StatementStatistics for slow query logging implementation ( #4719 )
...
* feat: log slow query
* feat: log slow query for sql
* refactor: add slow query logging options
* ci: fix errors
* feat: add StatementStatistics
* chore: revert modification of servers crate
* docs: update config docs
* fix: clippy errors
2024-09-30 03:26:50 +00:00
Lanqing Yang
cc4106cbd2
feat: protect datanode with concurrency limit. ( #4699 )
...
Adding parallelism in region server to protect datanode from query overload.
2024-09-27 06:12:57 +00:00
zyy17
aa03d3b11c
docs: use docs comment prefix and bump toml2docs version ( #4711 )
2024-09-11 07:49:23 +00:00
Lei, HUANG
208afe402b
feat(wal): increase recovery parallelism ( #4689 )
...
* Refactor RaftEngineLogStore to use references for config
- Updated `RaftEngineLogStore::try_new` to accept a reference to `RaftEngineConfig` instead of taking ownership.
- Replaced direct usage of `config` with individual fields (`sync_write`, `sync_period`, `read_batch_size`).
- Adjusted test cases to pass references to `RaftEngineConfig`.
* Add parallelism configuration for WAL recovery
- Introduced `recovery_parallelism` setting in `datanode.example.toml` and `standalone.example.toml` for configuring parallelism during WAL recovery.
- Updated `Cargo.lock` and `Cargo.toml` to include `num_cpus` dependency.
- Modified `RaftEngineConfig` to include `recovery_parallelism` with a default value set to the number of CP
* feat/wal-recovery-parallelism:
Add `wal.recovery_parallelism` configuration option
- Introduced `wal.recovery_parallelism` to config.md for specifying parallelism during WAL recovery.
- Updated `RaftEngineLogStore` to include `recovery_threads` from the new configuration.
* fix: ut
2024-09-09 04:25:24 +00:00
LFC
d43e31c7ed
feat: schedule compaction when adding sst files by editing region ( #4648 )
...
* feat: schedule compaction when adding sst files by editing region
* add minimum time interval for two successive compactions
* resolve PR comments
2024-09-04 10:10:07 +00:00
zyy17
8453df1392
refactor: make init_global_logging() clean and add log_format ( #4657 )
...
refactor: refine the code logic of init_global_logging and add json output format
2024-09-04 03:04:51 +00:00
dennis zhuang
a612b67470
feat: supports name in object storage config ( #4630 )
...
* feat: supports name in object storage config
* fix: integration test
* fix: integration test
* fix: update sample config
* fix: config api test
2024-09-03 07:02:55 +00:00
dennis zhuang
841e66c810
fix: config api and export metrics default database ( #4633 )
2024-08-28 14:28:49 +00:00
Yingwen
a12a905578
chore: disable ttl for write cache by default ( #4595 )
...
* chore: remove default write cache ttl
* docs: update example config
* chore: fix ci
2024-08-21 08:38:38 +00:00
Weny Xu
76dc906574
feat(log_store): introduce the CollectionTask ( #4530 )
...
* feat: introduce the `CollectionTask`
* feat: add config of index collector
* chore: remove unused code
* feat: truncate indexes
* chore: apply suggestions from CR
* chore: update config examples
* refactor: retrieve latest offset while dumping indexes
* chore: print warn
2024-08-19 03:48:35 +00:00
liyang
f578155602
feat: add GcsConfig credential field ( #4568 )
2024-08-16 03:11:20 +00:00
Weny Xu
2e2eacf3b2
feat: add SASL and TLS config for Kafka client ( #4536 )
...
* feat: add SASL and TLS config
* feat: add SASL/PLAIN and TLS config for Kafka client
* chore: use `ring`
* feat: support SASL SCRAM-SHA-256 and SCRAM-SHA-512
* fix: correct unit test
* test: add integration test
* chore: apply suggestions from CR
* refactor: introduce `KafkaConnectionConfig`
* chore: refine toml examples
* docs: add missing fields
* chore: refine examples
* feat: allow no server ca cert
* chore: refine examples
* chore: fix clippy
* feat: load system ca certs
* chore: fmt toml
* chore: unpin version
* Update src/common/wal/src/error.rs
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2024-08-12 12:27:11 +00:00
Weny Xu
a25d9f736f
chore: set default otlp_endpoint ( #4508 )
...
* chore: set default `otlp_endpoint`
* fix: fix ci
2024-08-06 06:48:14 +00:00
Ruihang Xia
7daf24c47f
feat: remove dedicated runtime for grpc, mysql and pg protocols ( #4436 )
...
* feat: remove dedicated runtime for grpc, mysql and pg protocols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove other runtimes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* spawn compact task into compact_runtime
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refine naming
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/servers/tests/mysql/mysql_server_test.rs
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* turnoff fuzz test matrix fail fast option
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update rt config for ci tests
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-07-30 06:17:58 +00:00
Weny Xu
7ad248d6f6
fix(config): enable file engine by default ( #4345 )
...
* fix: enable file engine by default
* fix: fix tests
2024-07-11 17:35:18 +00:00
Zhenchi
a710676d06
feat(fulltext_index): integrate full-text indexer with sst writer ( #4302 )
...
* feat(fulltext_index): integrate full-text indexer with sst writer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: delay building puffin writer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: indexer test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add abort on empty indexer
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: indicates default mode
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: introduce "auto" and "unlimited" as mem threshold
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* doc: comment about push empty string
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-07 04:10:19 +00:00
Zhenchi
65c9fbbd2f
feat(fulltext_index): integrate puffin manager with inverted index applier ( #4266 )
...
* feat(fulltext_index): integrate puffin manager with inverted index applier
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: get rid of unexpected not found from write cache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: move create_dir_all to BoundedStager::new
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: update config.md
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* config: unify directories
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: silent remove
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: config docs
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: auxiliary -> aux
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-04 06:18:58 +00:00
dennis zhuang
a779cb36ec
fix: wrong frontend registration address ( #4199 )
...
* fix: frontend registration address is wrong, #4186
* fix: license header
* chore: adds hostname to frontend grpc
* fix: forgot run make config-docs
* chore: warn when using bind_addr
* fix: flow node heartbeat carrying address
2024-06-26 06:13:07 +00:00
taobo
22d12683b4
refactor!: unify FrontendOptions and DatanodeOptions by using GrpcOptions ( #4088 )
...
* refactor: move GrpcOptions to servers/grpc
* fix: optimize code
* fix: docs
* refactor: move DatanodeOptions.rpc_hostname to grpc.hostname
* fix: merge main
* refactor code impl
test: add test_depreacted_cli_options unit test
* Update src/servers/src/grpc.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-18 22:45:38 +00:00
Weny Xu
ea2d067cf1
feat: implement the OrderedBatchProducer ( #4134 )
...
* feat: implement the `OrderedBatchProducer`
* test: add test of cancel safety
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: simplify the `BackgroundProducerWorker`
* feat: implement the OrderedBatchProducer v2
* refactor: switch to `OrderedBatchProducer`
* chore: rename to `MAX_FLUSH_QUEUE_SIZE`
* refactor: switch to `OrderedBatchProducerV2`
* refactor: remove `OrderedBatchProducerV1`
* test: add tests
* refactor: make config configurable
* refactor: minor refactor
* chore: remove unused code
* chore: remove `benchmarks` crate
* chore: update config doc
* chore: remove unused comment
* refactor: refactor client registry
* refactor: rename `max_batch_size` to `max_batch_bytes`
* chore: use constant value
* chore: ensure serialized meta < ESTIMATED_META_SIZE
* chore: apply suggestions from CR
* chore: remove the `CHANNEL_SIZE`
* chore: apply suggestions from CR
* fix: ensure serialized meta < ESTIMATED_META_SIZE
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-06-18 07:20:01 +00:00
Ning Sun
70d113a355
feat: update default size of bgworkers, add hbworkers ( #4129 )
...
* feat: update default size of bgworkers, add hbworkers
* feat: update frontend heartbeat as well
* chore: update sample config files and default settings
* chore: update config docs
* Revert "chore: update config docs"
This reverts commit 8107f4c120f5ae701c8d
2024-06-18 06:18:37 +00:00
Weny Xu
e142ca40d7
feat: invoke handle_batch_open_requests ( #4107 )
...
* feat: open all regions via invoking `handle_batch_open_requests`
* tests: add sqlness tests
* refactor: avoid cloning
* chore: apply suggestions from CR
* chore: update config.md
* chore: apply suggestions from CR
2024-06-07 09:07:45 +00:00
LFC
c0aed1d267
feat: set global runtime size by config file ( #4063 )
...
* set global runtime size
* fix: resolve PR comments
* fix: log the whole option
* fix ci
* debug ci
* debug ci
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-04 10:03:33 +00:00
shuiyisong
9e1af79637
chore: add ttl to write_cache ( #4010 )
...
* chore: add ttl to write_cache
* chore: update test & add example config
* chore: fix typo
* chore: fix typo
* chore: fix typo
2024-05-22 06:50:12 +00:00
zyy17
60eb5de3f1
refactor: add tracing options in xOptions ( #3919 )
...
* refactor: add tracing options in {DatanodeOptions, FrontendOptions, MetasrvOptions, StandaloneOptions}
* ci: fix integration error
* refactor: minor modification for initialization of tracing options
2024-05-13 05:16:50 +00:00
Yingwen
d2e081c1f9
docs: update memtable config example ( #3712 )
2024-04-16 07:26:20 +00:00
zyy17
8060c81e1d
refactor: use toml2docs to generate config docs ( #3704 )
...
* refactor: use toml2docs to generate config docs
* ci: add docs check in 'check-typos-and-docs'
2024-04-15 09:08:32 +00:00
Lei, HUANG
183fccbbd6
chore: remove global_ttl config ( #3673 )
...
* chore: remove global_ttl config
* fix: clippy
2024-04-09 02:00:50 +00:00
Yingwen
39b69f1e3b
refactor!: Renames the new memtable to PartitionTreeMemtable ( #3547 )
...
* refactor: rename mod merge_tree to partition_tree
* refactor: rename merge_tree
* refactor: change merge tree comment
* refactor: rename merge tree struct
* refactor: memtable options
2024-03-20 06:40:41 +00:00
Lei, HUANG
7183fa198c
refactor: make MergeTreeMemtable the default choice ( #3430 )
...
* refactor: make MergeTreeMemtable the default choice
* refactor: reformat
* chore: add doc to config
2024-03-05 10:00:08 +00:00
Zhenchi
c3c80b92c8
feat(index): measure memory usage in global instead of single-column and add metrics ( #3383 )
...
* feat(index): measure memory usage in global instead of single-column and add metrics
* feat: add leading zeros to streamline memory usage
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: fmt
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: remove println
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-02-28 06:49:24 +00:00
Weny Xu
19413eb345
refactor!: rename initialize_region_in_background to init_regions_in_background ( #3216 )
...
refactor!: change initialize_region_in_background to init_regions_in_background
2024-01-23 04:33:48 +00:00
shuiyisong
2bf4b08a6b
chore: change default factor to compute memory size ( #3211 )
...
* chore: change default factor to compute memory size
* chore: update doc
* chore: update comment in example config
* chore: extract factor to const and update comments
* chore: update comment by cr suggestion
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 09:03:29 +00:00
Zhenchi
6f07d69155
feat(mito): enable inverted index ( #3158 )
...
* feat(mito): enable inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* accidentally resolved the incorrect filtering issue within the Metric Engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
* Update src/mito2/src/test_util/scheduler_util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: format -> join_dir
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move intermediate_manager from arg of write_and_upload_sst to field of WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: add IndexerBuidler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 09:08:07 +00:00
niebayes
f5798e2833
fix: remove incorrect wal comments in config file ( #3142 )
...
fix: kafka config comments
2024-01-11 09:34:24 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
niebayes
78303639db
feat(remote_wal): split an entry if it's too large ( #3092 )
...
* feat: split an entry if it's too large
* chore: rewrite check records
* test: add some unit tests for record
* chore: rewrite entry splitting
* chore: add unit tests for build records
* chore: add more unit tests for record
* chore: rewrite encdec of record
* revert: ignored test
* fix: set limit for max_batch_size
* fix: clippy
* chore: remove heavy logging
* fix: CR
* fix: properly terminate
* fix: CR
* fix: compiling
* fix: sqlness
* fix: CR
* fix: license
* fix: license
2024-01-05 12:41:43 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
Weny Xu
5653389063
feat!: correct the kafka config option ( #3065 )
...
* feat: correct the kafka config option
* refactor: rewrite the verbose comments
2024-01-02 07:31:37 +00:00
niebayes
d061bf3d07
feat(remote_wal): introduce kafka remote wal ( #3001 )
...
* feat: integrate remote wal to standalone
* fix: test
* chore: ready to debug kafka remote wal
* fix: test
* chore: add some logs for remote wal
* chore: add logs for topic manager
* fix: properly terminate stream consumer
* fix: properly handle TopicAlreadyExists error
* fix: parse config file error
* fix: properly handle last entry id
* chore: prepare for merge
* fix: test
* fix: typo
* fix: set replication_factor properly
* fix: CR
* test: tmp for test
* Revert "test: tmp for test"
This reverts commit 093a3e0038
2023-12-26 12:35:24 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
niebayes
bab198ae68
feat(remote_wal): impl kafka log store ( #2971 )
...
* feat: introduce client manager
* chore: add errors for client manager
* chore: add record utils
* chore: impl kafka log store
* chore: build kafka log store upon starting datanode
* chore: update comments for kafka log store
* chore: add a todo for getting entry offset
* fix: typo
* chore: remove unused
* chore: update comments
* fix: typo
* fix: resolve some review conversations
* chore: move commonly referenced crates to workspace Cargo.toml
* fix: style
* fix: style
* chore: unify topic name prefix
* chore: make backoff config configurable by users
* chore: properly use backoff config in wal config
* refactor: read/write of kafka log store
* fix: typo
* fix: typo
* fix: resolve review conversations
2023-12-25 09:21:52 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
