Weny Xu
4045298cb2
feat: add region_statistics table ( #4771 )
...
* refactor: introduce `region_statistic`
* refactor: move DatanodeStat related structs to common_meta
* chore: add comments
* feat: implement `list_region_stats` for `ClusterInfo` trait
* feat: add `region_statistics` table
* feat: add table_id and region_number fields
* chore: rename unused snafu
* chore: udpate sqlness results
* chore: avoid to print source in error msg
* chore: move `procedure_info` under `greptime` catalog
* chore: apply suggestions from CR
* Update src/common/meta/src/datanode.rs
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
---------
Co-authored-by: jeremyhi <jiachun_feng@proton.me >
2024-09-27 09:54:52 +00:00
Lanqing Yang
cc4106cbd2
feat: protect datanode with concurrency limit. ( #4699 )
...
Adding parallelism in region server to protect datanode from query overload.
2024-09-27 06:12:57 +00:00
Weny Xu
163cea81c2
feat: migrate local WAL regions ( #4715 )
...
* feat: allow to flush region before migrating
* fix: fix unit tests
* feat: allow to set `flush_timeout`
* feat: skip to replay memtable
* fix: fix unit tests
* test: add more tests
* refactor: simplify timeout logical
* test: add unit tests
* test: add unit tests
* chore: update comments
* fix: fix unit tests
* fix: fmt and clippy
* feat: change default timeout to 30s
* fix: throw `ExceededDeadline` error
* test: add tests for `downgrade_region_with_retry`
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: update proto to `3633474`
* refactor: refactor `upgrade_region_with_retry`
* chore: apply suggestions from CR
2024-09-20 08:27:20 +00:00
Ruihang Xia
f5cf25b0db
refactor: remove DfPlan wrapper ( #4733 )
...
* refactor: remove DfPlan wrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test assertion
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-19 12:29:33 +00:00
jeremyhi
dcae21208b
chore: refresh route table ( #4673 )
...
* chore: remove error::
* chore: avoid to use get_raw if unnecessary
* chore: clearer method name
* feat: remap node addresses in table route
* chore: add unit test for remap address
* feat: refresh node address mapping via heartbeat
* feat: broadcast table cache invalidate on new epoch
* chore: clarify heartbeat log
* chore: remove InvalidHeartbeatRequest
* chore: add log
* feat: add role into NodeAddressKey
* chore: fix test
* Update src/common/meta/src/key/table_route.rs
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
* chore: simplify code
---------
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
2024-09-10 12:08:59 +00:00
Lei, HUANG
208afe402b
feat(wal): increase recovery parallelism ( #4689 )
...
* Refactor RaftEngineLogStore to use references for config
- Updated `RaftEngineLogStore::try_new` to accept a reference to `RaftEngineConfig` instead of taking ownership.
- Replaced direct usage of `config` with individual fields (`sync_write`, `sync_period`, `read_batch_size`).
- Adjusted test cases to pass references to `RaftEngineConfig`.
* Add parallelism configuration for WAL recovery
- Introduced `recovery_parallelism` setting in `datanode.example.toml` and `standalone.example.toml` for configuring parallelism during WAL recovery.
- Updated `Cargo.lock` and `Cargo.toml` to include `num_cpus` dependency.
- Modified `RaftEngineConfig` to include `recovery_parallelism` with a default value set to the number of CP
* feat/wal-recovery-parallelism:
Add `wal.recovery_parallelism` configuration option
- Introduced `wal.recovery_parallelism` to config.md for specifying parallelism during WAL recovery.
- Updated `RaftEngineLogStore` to include `recovery_threads` from the new configuration.
* fix: ut
2024-09-09 04:25:24 +00:00
jeremyhi
e88465840d
feat: add extension field to HeartbeatRequest ( #4688 )
...
* feat: add extension field to HeartbeatRequest
* chore: extension to extensions
* chore: upgrade proto
2024-09-06 08:29:20 +00:00
Ruihang Xia
93f202694c
refactor: remove unused error variants ( #4666 )
...
* add python script
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unused errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix all negative cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* setup CI
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add license header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-09-03 13:19:38 +00:00
dennis zhuang
a612b67470
feat: supports name in object storage config ( #4630 )
...
* feat: supports name in object storage config
* fix: integration test
* fix: integration test
* fix: update sample config
* fix: config api test
2024-09-03 07:02:55 +00:00
Yingwen
8eda36bfe3
feat: remove files from the write cache in purger ( #4655 )
...
* feat: remove files from the write cache in purger
* chore: fix typo
2024-08-31 04:19:52 +00:00
Lanqing Yang
9286e963e7
chore: adding heartbeat sent/recv counts in greptimedb nodes ( #4624 )
...
obs: adding heartbeat sent/recv counts in greptimedb nodes
2024-08-30 03:57:16 +00:00
Weny Xu
47657ebbc8
feat: replay WAL entries respect index ( #4565 )
...
* feat(log_store): use new `Consumer`
* feat: add `from_peer_id`
* feat: read WAL entries respect index
* test: add test for `build_region_wal_index_iterator`
* fix: keep the handle
* fix: incorrect last index
* fix: replay last entry id may be greater than expected
* chore: remove unused code
* chore: apply suggestions from CR
* chore: rename `datanode_id` to `location_id`
* chore: rename `from_peer_id` to `location_id`
* chore: rename `from_peer_id` to `location_id`
* chore: apply suggestions from CR
2024-08-28 11:37:18 +00:00
dennis zhuang
63f2463273
feat!: impl admin command ( #4600 )
...
* feat: impl admin statement parser
* feat: introduce AsyncFunction and implements it for admin functions
* feat: execute admin functions
* fix: license header
* fix: panic in test
* chore: fixed by code review
2024-08-26 07:53:40 +00:00
Weny Xu
76dc906574
feat(log_store): introduce the CollectionTask ( #4530 )
...
* feat: introduce the `CollectionTask`
* feat: add config of index collector
* chore: remove unused code
* feat: truncate indexes
* chore: apply suggestions from CR
* chore: update config examples
* refactor: retrieve latest offset while dumping indexes
* chore: print warn
2024-08-19 03:48:35 +00:00
liyang
f578155602
feat: add GcsConfig credential field ( #4568 )
2024-08-16 03:11:20 +00:00
zyy17
63e1892dc1
refactor(plugin): add SetupPlugin and StartPlugin error ( #4554 )
2024-08-13 11:22:48 +00:00
Weny Xu
cb4cffe636
chore: bump opendal version to 0.48 ( #4499 )
2024-08-04 00:46:04 +00:00
Ruihang Xia
7daf24c47f
feat: remove dedicated runtime for grpc, mysql and pg protocols ( #4436 )
...
* feat: remove dedicated runtime for grpc, mysql and pg protocols
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove other runtimes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* spawn compact task into compact_runtime
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* refine naming
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/servers/tests/mysql/mysql_server_test.rs
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* turnoff fuzz test matrix fail fast option
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update rt config for ci tests
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-07-30 06:17:58 +00:00
Weny Xu
164232e073
fix: use heartbeat runtime instead of background runtime ( #4445 )
2024-07-29 03:29:30 +00:00
discord9
021ec7b6ac
feat(flow): flush_flow function ( #4416 )
...
* refactor: df err variant
* WIP
* chore: update proto version
* chore: revert mistaken rust-toolchain
* feat(WIP): added FlowService to QueryEngine
* refactor: move flow service to operator
* refactor: flush use flow name not id
* refactor: use full path in macro
* feat: flush flow
* feat: impl flush flow
* chore: remove unused
* chore: meaninful response
* chore: remove unused
* chore: clippy
* fix: flush_flow with proper blocking
* test: sqlness tests added back for flow
* test: better predicate for flush_flow
* refactor: rwlock
* fix: flush lock
* fix: flush lock write then drop
* test: add a new flow sqlness test
* fix: sqlness testcase
* chore: style
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-07-26 23:04:13 +00:00
zyy17
b81d3a28e6
refactor: add RetryInterceptor to print detailed error ( #4434 )
2024-07-25 11:52:28 +00:00
dennis zhuang
64cad4e891
feat: tweak error and status codes ( #4359 )
...
* feat: tweak status codes
* fix: typo
* fix: by cr comments
2024-07-15 07:50:16 +00:00
Ning Sun
b8bd8456f0
fix: remove path label for cache store ( #4336 )
...
* fix: remove path label for cache store
* fix: ignore path label for intermediatemanager
* refactor: remove unused object store
2024-07-15 03:34:19 +00:00
Jeremyhi
ddf2e6a3c0
feat: provide a simple way to create metaclient ( #4257 )
...
* feat: provide a simple way to create metaclient
* chore: minor refactor using metaclient
* chore: minor refactor using metaclient
2024-07-03 08:11:55 +00:00
zyy17
f2c08b8ddd
feat: introduce the interface of RemoteJobScheduler ( #4181 )
...
* refactor: add Compactor trait
* chore: add compact() in Compactor trait and expose compaction module
* refactor: add CompactionRequest and open_compaction_region
* refactor: export the compaction api
* refactor: add DefaultCompactor::new_from_request
* refactor: no need to pass mito_config in open_compaction_region()
* refactor: CompactionRequest -> &CompactionRequest
* fix: typo
* docs: add docs for public apis
* refactor: remove 'Picker' from Compactor
* chore: add logs
* chore: change pub attribute for Picker
* refactor: remove do_merge_ssts()
* refactor: update comments
* refactor: use CompactionRegion argument in Picker
* chore: make compaction module public and remove unnessary clone
* refactor: move build_compaction_task() in CompactionScheduler{}
* chore: use in open_compaction_region() and add some comments for public structure
* refactor: add 'manifest_dir()' in store-api
* refactor: move the default implementation to DefaultCompactor
* refactor: remove Options from MergeOutput
* chore: minor modification
* fix: clippy errors
* fix: unit test errors
* refactor: remove 'manifest_dir()' from store-api crate(already have one in opener)
* refactor: use 'region_dir' in CompactionRequest
* refactor: refine naming
* refactor: refine naming
* refactor: remove clone()
* chore: add comments
* refactor: add PickerOutput field in CompactorRequest
* feat: introduce RemoteJobScheduler
* feat: add RemoteJobScheudler in schedule_compaction_request()
* refactor: use Option type for senders field of CompactionFinished
* refactor: modify CompactionJob
* refactor: schedule remote compaction job by options
* refactor: remove unused Options
* build: remove unused log
* refactor: fallback to local compaction if the remote compaction failed
* fix: clippy errors
* refactor: add plugins in mito2
* refactor: add from_u64() for JobId
* refactor: make schedule module public
* refactor: add error for RemoteJobScheduler
* refactor: add Notifier
* refactor: use Arc for Notifier
* refactor: add 'remote_compaction' in compaction options
* fix: clippy errors
* fix: unrecognized table option
* refactor: add 'start_time' in CompactionJob
* refactor: modify error type of RemoteJobScheduler
* chore: revert changes for request
* refactor: code refactor by review comment
* refactor: use string type for JobId
* refactor: add 'waiters' field in DefaultNotifier
* fix: build error
* refactor: take coderabbit's review comment
* refactor: use uuid::Uuid as JobId
* refactor: return waiters when schedule failed and add on_failure for DefaultNotifier
* refactor: move waiters from notifier to Job
* refactor: use ObjectStoreManagerRef in open_compaction_region()
* refactor: implement for JobId and adds related unit tests
* fix: run unit tests failed
* refactor: add RemoteJobSchedulerError
2024-07-02 07:08:43 +00:00
tison
2665616f72
build(deps): Upgrade OpenDAL to 0.47 ( #4224 )
...
* catch up changes
Signed-off-by: tison <wander4096@gmail.com >
* fmt
Signed-off-by: tison <wander4096@gmail.com >
* Fix cache for 0471 (#7 )
* Fix cache for 0471
Signed-off-by: Xuanwo <github@xuanwo.io >
* Make clippy happy
Signed-off-by: Xuanwo <github@xuanwo.io >
---------
Signed-off-by: Xuanwo <github@xuanwo.io >
* tidy
Signed-off-by: tison <wander4096@gmail.com >
* use opendal's exported type
Signed-off-by: tison <wander4096@gmail.com >
* clippy
Signed-off-by: tison <wander4096@gmail.com >
* fmt
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Signed-off-by: Xuanwo <github@xuanwo.io >
Co-authored-by: Xuanwo <github@xuanwo.io >
2024-07-01 17:05:15 +00:00
zyy17
fe2c5c3735
refactor: expose DatanodeBuilder::build_object_store_manager() and MitoConfig::sanitize() ( #4212 )
...
* refactor: expose DatanodeBuilder::build_object_store_manager()
* refactor: expose MitoConfig::sanitize()
2024-07-01 06:36:32 +00:00
Jeremyhi
214fd38f69
feat: add build info for flow heartbeat task ( #4228 )
...
* chore: refactor load region stats
* feat: add build info for flow heartbeat
2024-07-01 03:19:25 +00:00
dennis zhuang
a779cb36ec
fix: wrong frontend registration address ( #4199 )
...
* fix: frontend registration address is wrong, #4186
* fix: license header
* chore: adds hostname to frontend grpc
* fix: forgot run make config-docs
* chore: warn when using bind_addr
* fix: flow node heartbeat carrying address
2024-06-26 06:13:07 +00:00
Weny Xu
b2f61aa1cf
fix: format error correctly ( #4204 )
...
* chore: remove TODO comments
* fix: format error correctly
2024-06-25 07:56:13 +00:00
Weny Xu
15ad9f2f6f
fix: region logical regions after catching up ( #4176 )
...
* fix: region logical regions after catching up
* test: add metric table migration test
* chore: apply suggestions from CR
2024-06-21 10:30:18 +00:00
taobo
22d12683b4
refactor!: unify FrontendOptions and DatanodeOptions by using GrpcOptions ( #4088 )
...
* refactor: move GrpcOptions to servers/grpc
* fix: optimize code
* fix: docs
* refactor: move DatanodeOptions.rpc_hostname to grpc.hostname
* fix: merge main
* refactor code impl
test: add test_depreacted_cli_options unit test
* Update src/servers/src/grpc.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-18 22:45:38 +00:00
Ning Sun
70d113a355
feat: update default size of bgworkers, add hbworkers ( #4129 )
...
* feat: update default size of bgworkers, add hbworkers
* feat: update frontend heartbeat as well
* chore: update sample config files and default settings
* chore: update config docs
* Revert "chore: update config docs"
This reverts commit 8107f4c120 .
* Revert "chore: update sample config files and default settings"
This reverts commit f5ae701c8d .
* feat: use default heartbeat runtime size
* chore: update config docs
2024-06-18 06:18:37 +00:00
Jeremyhi
141d017576
feat: enable metasrv to accept flownode's heartbeats ( #4160 )
...
* feat: make metasrv can accept flownode's heartbeat
* chore: proto
2024-06-18 04:07:46 +00:00
LFC
5a6021e34f
refactor: remove substrait ser/de for region query in standalone ( #3812 )
...
* refactor: remove substrait serde for region query in standalone
* fix ci
* move QueryRequest to common-query
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format toml file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: format toml
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-11 09:33:47 +00:00
Weny Xu
e142ca40d7
feat: invoke handle_batch_open_requests ( #4107 )
...
* feat: open all regions via invoking `handle_batch_open_requests`
* tests: add sqlness tests
* refactor: avoid cloning
* chore: apply suggestions from CR
* chore: update config.md
* chore: apply suggestions from CR
2024-06-07 09:07:45 +00:00
LFC
c0aed1d267
feat: set global runtime size by config file ( #4063 )
...
* set global runtime size
* fix: resolve PR comments
* fix: log the whole option
* fix ci
* debug ci
* debug ci
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-04 10:03:33 +00:00
shuiyisong
d1838fb28d
refactor: move define_into_tonic_status to common-error ( #4095 )
...
* chore: finish cherry-pick
* chore: remove unused code
2024-06-04 03:29:15 +00:00
Ruihang Xia
45fee948e9
fix: display error in correct format ( #4082 )
...
* fix: display error in correct format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add address to RegionServer error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-31 09:25:14 +00:00
dennis zhuang
24612f62dd
feat: querying from view works ( #3952 )
...
* feat: querying from view works
* feat: use MemoryCatalogProviderList instead of DummyCatalogList
* refactor: revert src/query/src/dummy_catalog.rs
* chore: clean code
* fix: make clippy happy
* fix: toml format
* fix: sqlness
* fix: forgot files
* fix: make sqlness happy
* test: table source, serializer and decoder
* fix: fail to decode plan because of invalid table names
* test: adds more sqlness test for view
* chore: remove unused errors
* fix: comments
* fix: typo
* fix: invalidate view info cache after creating view successfully
* chore: apply suggestion
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: apply suggestion
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: compile error after rebeasing
* chore: style
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* fix: don't export table_name in common-meta
* chore: change ViewInfo::new signature
* docs: leave a TODO for optimize param
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-05-30 21:45:56 +00:00
Weny Xu
eab309ff7e
fix: avoid acquiring lock during reading stats ( #4070 )
...
* fix: avoid acquiring lock during reading stats
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-05-30 07:08:04 +00:00
Weny Xu
4aa756c896
feat: open region in background ( #4052 )
...
* feat: open region in background
* feat: trace opening regions
* feat: wait for the opening region
* feat: let engine to handle the future open request
* fix: fix `test_region_registering`
2024-05-28 13:58:15 +00:00
Weny Xu
d3860671a8
chore: add LAST_SENT_HEARTBEAT_ELAPSED metric ( #4062 )
2024-05-28 08:05:37 +00:00
tison
f9db5ff0d6
build(deps): upgrade opendal to 0.46 ( #4037 )
...
* build(deps): upgrade opendal to 0.46
Signed-off-by: tison <wander4096@gmail.com >
* migrate writes
Signed-off-by: tison <wander4096@gmail.com >
* migrate reads
Signed-off-by: tison <wander4096@gmail.com >
* fixup object safety
Signed-off-by: tison <wander4096@gmail.com >
* fixup names
Signed-off-by: tison <wander4096@gmail.com >
* fixup compilation
Signed-off-by: tison <wander4096@gmail.com >
* fixup compilation
Signed-off-by: tison <wander4096@gmail.com >
* a few Buffer to Vec
Signed-off-by: tison <wander4096@gmail.com >
* Make greptime buildable with opendal 0.46 (#5 )
Signed-off-by: Xuanwo <github@xuanwo.io >
* fixup toml check
Signed-off-by: tison <wander4096@gmail.com >
* test_orc_opener
Signed-off-by: tison <wander4096@gmail.com >
* Fix lru cache (#6 )
Signed-off-by: Xuanwo <github@xuanwo.io >
* clippy
Signed-off-by: tison <wander4096@gmail.com >
* improve comments
Signed-off-by: tison <wander4096@gmail.com >
* address comments
Signed-off-by: tison <wander4096@gmail.com >
* reduce buf copy
Signed-off-by: tison <wander4096@gmail.com >
* upgrade to reqwest 0.12
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Signed-off-by: Xuanwo <github@xuanwo.io >
Co-authored-by: Xuanwo <github@xuanwo.io >
2024-05-27 09:12:23 +00:00
LFC
297105266b
feat: enable tcp keepalive for http server ( #4019 )
...
* feat: enable tcp keepalive for http server
* chore: for enterprise's update
* resolve PR comments
2024-05-27 04:07:36 +00:00
taobo
a3a2c8d063
feat: Add TLS support for gRPC service ( #3957 )
...
* feat: Add tls support for grpc service
* feat: add integration test
* fix: integration test
* fix: revert by suggestion
* fix: typos
* fix: optimize code
* fix: optimize code
* docs: update configs
2024-05-23 19:00:16 +00:00
Jeremyhi
418090b464
chore: log error for detail ( #4011 )
...
* chore: log error for detail
* chore: by cr
2024-05-22 12:17:20 +00:00
Lei, HUANG
090b59e8d6
feat: manual compaction ( #3988 )
...
* add compaction udf params
* wip: pass compaction options through grpc
* wip: pass compaction options all the way down to region server
* wip: window compaction task
* feat: trigger major compaction
* refactor: optimize compaction parameter parsing
* chore: rebase main
* chore: update proto
* chore: add some tests
* feat: validate catalog
* chore: fix typo and rebase main
* fix: some cr comments
* fix: file_time_bucket_span
* fix: avoid upper bound overflow
* chore: update proto
2024-05-22 09:42:21 +00:00
Yingwen
179c8c716c
feat: Adds RegionScanner trait ( #3948 )
...
* feat: define region scanner
* feat: single partition scanner
* feat: use single partition scanner
* feat: implement ExecutionPlan wip
* feat: mito engine returns single partition scanner
* feat: implement DisplayAs for region server
* feat: dummy table provider use handle_partitioned_query()
* test: update sqlness test
* feat: table provider use ReadFromRegion
* refactor: remove StreamScanAdapter
* chore: update lock
* style: fix clippy
* refactor: remove handle_query from the RegionEngine trait
* chore: address CR comments
* refactor: rename methods
* refactor: rename ReadFromRegion to RegionScanExec
2024-05-20 11:52:00 +00:00
Weny Xu
f696f41a02
fix: prevent registering logical regions with AliveKeeper ( #3965 )
...
* fix: register logical region
* chore: fix Clippy
* chore: apply suggestions from CR
2024-05-17 07:38:35 +00:00