fys
6e860bc0fd
feat: support grpc for otlp trace and metrics ( #3105 )
...
* feat: add grpc support for otlp trace and metrics
* cr: add some comment
* fix: ut
* fix: cr
2024-01-09 05:01:48 +00:00
Yingwen
8bd4a36136
feat(mito): Init the write cache in datanode ( #3100 )
...
* feat: add builder to build cache manager
* refactor: make MitoEngine::new async
* refactor: refactor object store creation
* refactor: add helper fn to attaches layers
* feat: fn to build fs store
* feat: add write cache to engine
* feat: config write cache
* style: fix clippy
* test: fix test
* feat: add warning
* chore: add experimental prefix to configs
* test: fix config test
* test: test weighted size
* feat: add switch to enable write cache
* fix: update cache stats by using get
* style: use then
2024-01-09 04:40:22 +00:00
zyy17
a89840f5f9
refactor(metrics): add 'greptime_' prefix for every metrics ( #3093 )
...
* refactor(metrics): add 'greptimedb_' prefix for every metrics
* chore: use 'greptime_' as prefix
* chore: add some prefix for new metrics
* chore: fix format error
2024-01-05 08:12:23 +00:00
LFC
e0525dbfeb
chore: expose some codes to let other projects use them ( #3102 )
2024-01-05 06:54:01 +00:00
Weny Xu
cdc9021160
feat(metric): implement role and region_disk_usage ( #3095 )
...
* feat(metric): implement `role` and `region_disk_usage`
* Update src/datanode/src/region_server.rs
* Update src/datanode/src/heartbeat.rs
---------
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
2024-01-05 06:53:52 +00:00
Weny Xu
284a496f54
feat: add logs for upgrading candidate region and updating metadata ( #3077 )
...
* feat: add logs for upgrading candidate region
* feat: add logs for update metadata
* chore: apply suggestions from CR
2024-01-04 06:57:07 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
LFC
ec43b9183d
feat: table route for metric engine ( #3053 )
...
* feat: table route for metric engine
* feat: register logical regions
* fix: open logical region (#96 )
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-04 06:30:17 +00:00
Weny Xu
611a8aa2fe
feat(tests-integration): add a naive region migration integration test ( #3078 )
...
* fix: fix heartbeat handler ignore upgrade candidate instruction
* fix: fix handler did not inject wal options
* feat: expose `RegionMigrationProcedureTask`
* feat(tests-integration): add a naive region migration test
* chore: apply suggestions from CR
* feat: add test if the target region has migrated
* chore: apply suggestions from CR
2024-01-03 07:12:59 +00:00
Weny Xu
2b181e91e0
refactor: unify the injection of WAL option ( #3066 )
...
* feat: add prepare_wal_option
* refactor: use integer hashmap
* feat: unify the injection of WAL option
* fix: fix procedure_flow_upgrade_candidate_with_retry
* chore: apply suggestions from CR
2024-01-02 07:40:02 +00:00
Weny Xu
485a91f49a
feat: implement handle upgrade region instruction ( #3013 )
...
* feat: implement task tracker
* feat: implement handle upgrade region instruction
* refactor: remove redundant code
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: refactor wait_for_replay_millis to wait_for_replay_timeout
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-12-28 02:08:47 +00:00
niebayes
bab198ae68
feat(remote_wal): impl kafka log store ( #2971 )
...
* feat: introduce client manager
* chore: add errors for client manager
* chore: add record utils
* chore: impl kafka log store
* chore: build kafka log store upon starting datanode
* chore: update comments for kafka log store
* chore: add a todo for getting entry offset
* fix: typo
* chore: remove unused
* chore: update comments
* fix: typo
* fix: resolve some review conversations
* chore: move commonly referenced crates to workspace Cargo.toml
* fix: style
* fix: style
* chore: unify topic name prefix
* chore: make backoff config configurable by users
* chore: properly use backoff config in wal config
* refactor: read/write of kafka log store
* fix: typo
* fix: typo
* fix: resolve review conversations
2023-12-25 09:21:52 +00:00
Weny Xu
06fd7fd210
feat: add skip_wal_replay to OpenRegion instruction ( #2977 )
...
feat: add skip_wal_replay to OpenRegion instruction
2023-12-23 06:42:21 +00:00
niebayes
7d509e97f6
chore: move some commonly referenced crates to workspace Cargo.toml ( #2981 )
...
fix: resolve conflicts
2023-12-22 09:13:18 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
ff8c10eae7
feat: add CatchupRequest to engine ( #2939 )
...
* chore: remove redundant code
* feat(mito): add CatchupRequest
feat: reopen region before replay if need
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-12-21 08:27:53 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Weny Xu
97c3755ab6
feat(mito): add skip_wal_replay option to OpenRegionRequest ( #2955 )
...
* feat(mito): add skip_replay_wal option to OpenRegionRequest
* test: add tests for skip replay wal
* chore: rename `skip_replay_wal` to `skip_wal_replay`
2023-12-20 03:31:48 +00:00
niebayes
839e653e0d
feat(remote_wal): add skeleton for remote wal related to meta srv ( #2933 )
...
* feat: introduce wal config and kafka config
* feat: introduce kafka topic manager and selector
* feat: introduce region wal options
* chore: build region wal options upon starting meta srv
* feat: integrate region wal options allocator into table meta allocator
* chore: add wal config to metasrv.example.toml
* chore: add region wal options map to create table procedure
* feat: augment region create request with wal options
* feat: augment DatanodeTableValue with region wal options map
* chore: encode region wal options upon constructing table creator
* feat: persist region wal options when creating table meta
* fix: sqlness test
* chore: set default wal provider to raft-engine
* refactor: refactor wal options
* chore: update wal options allocator
* refactor: rename region wal options to wal options
* chore: update usages of region wal options
* chore: add some comments to kafka
* chore: fill in kafka config
* test: add tests for serde wal config
* test: add tests for wal options
* refactor: refactor wal options allocator to enum
* refactor: store wal options into the request options instead
* fix: typo
* fix: typo
* refactor: move wal options map to region info
* refactor: refacto serialization and deserialization of wal options
* refactor: use serde_json to encode wal options
* chore: rename wal_options_map to region_wal_options
* chore: resolve some review comments
* fix: typo
* refactor: replace kecab-case with snake_case
* fix: sqlness and converage tests
* fix: typo
* fix: coverage test
* fix: coverage test
* chore: resolve some review conversations
* fix: resolve some review conversations
* chore: format comments in metasrv.example.toml
* chore: update import style
* feat: integrate wal options allocator to standalone mode
* test: add compatible test for OpenRegion
* test: add compatible test for UpdateRegionMetadata
* chore: remove based suffix from topic selector type
2023-12-19 12:43:47 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
Zhenchi
029ff2f1e3
feat(inverted_index.create): add read/write for external intermediate files ( #2942 )
...
* feat(inverted_index.create): add read/write for external intermediate files
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: fix typos intermedia -> intermediate
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: futures_code -> asynchronous_codec
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: bump bytes to 1.5
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 09:44:48 +00:00
Ruihang Xia
9af9c0229a
feat: create table procedure for metric engine, part 1 ( #2943 )
...
* implementation
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* initialize
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove empty file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply review sugg
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-18 08:03:28 +00:00
LFC
181e16a11a
refactor: make instance started separately ( #2911 )
...
* refactor: make instance started separately, to support further integrated into other binaries
* fix: resolve PR comments
* fix: resolve PR comments
2023-12-14 03:44:29 +00:00
Yue Deng
3d651522c2
feat: add build() function to return the database build info ( #2919 )
...
* feat: add build function and register it
build() function to return the database build info #2909
* refactor: fix typos and change code structure
* test: add test for build()
* refactor: cargo fmt and eliminate warnings
* Apply suggestions from code review
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* refactor: move system.sql to a new directory
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-12-13 09:02:00 +00:00
Weny Xu
cf6bba09fd
refactor: use downgrading the region instead of closing region ( #2863 )
...
* refactor: use downgrading the region instead of closing region
* feat: enhance the tests for alive keeper
* feat: add a metric to track region lease expired
* chore: apply suggestions from CR
* chore: enable logging for test_distributed_handle_ddl_request
* refactor: simplify lease keeper
* feat: add metrics for lease keeper
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move OpeningRegionKeeper to common_meta
* feat: register operating regions to MemoryRegionKeeper
2023-12-12 09:24:17 +00:00
dennis zhuang
cf8b6c77dc
chore: clean up unused errors ( #2901 )
2023-12-11 09:08:50 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
tison
f74715ce52
refactor: RegionEngine::handle_request always returns affected rows ( #2874 )
...
* refactor: RegionEngine::handle_request -> handle_execution
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* revert spell change
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo clippy
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo fmt
Signed-off-by: tison <wander4096@gmail.com >
* more name clarification
Signed-off-by: tison <wander4096@gmail.com >
* revert rename
Signed-off-by: tison <wander4096@gmail.com >
* wrap affected rows into RegionResponse
Signed-off-by: tison <wander4096@gmail.com >
* flatten return AffectedRows
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-12-06 13:27:19 +00:00
WU Jingdi
aa89d9deef
fix: replace opendal PrometheusLayer ( #2861 )
...
* fix: replace opendal `PrometheusLayer`
* chore: add docs on `PrometheusMetricsLayer`
* chore: fix code advice
* chore: fix bug on `PrometheusMetricsLayer`
2023-12-05 07:15:45 +00:00
Weny Xu
b3ffe5cd1e
feat: handle the downgrade region instruction ( #2855 )
...
* feat: handle the downgrade region instruction
* test: add tests for RegionHeartbeatResponseHandler
* refactor: remove unused code
2023-12-05 03:30:55 +00:00
Weny Xu
7a14db68a6
feat: add upgrade candidate region step ( #2829 )
...
* feat: add upgrade candidate region step
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-12-04 05:09:27 +00:00
Weny Xu
c26f2f94c0
chore: add logs and metrics ( #2858 )
...
* chore: add logs and metrics
* feat: add the timer to track heartbeat intervel
* feat: add the gauge to track region leases
* refactor: use gauge instead of the timer
* chore: apply suggestions from CR
* feat: add hit rate and etcd txn metrics
2023-12-04 02:51:30 +00:00
Weny Xu
cce5edc88e
feat: add downgrade leader region step ( #2792 )
...
* feat: add downgrade leader region step
* chore: apply suggestions from CR
* chore: rename exist to exists
* chore: apply suggestions from CR
2023-11-29 09:17:28 +00:00
Weny Xu
9e58bba363
feat: add set_readonly_gracefully for engine ( #2787 )
...
* feat: add set_readonly_gracefully for engine
* chore: apply suggestions from CR
* chore: rename to set_readonly_test
* chore: apply suggestions from CR
2023-11-24 10:59:05 +00:00
LFC
64a36e9b36
refactor: start datanode more flexibly ( #2800 )
...
* refactor: start datanode more flexibly
* Update src/datanode/src/datanode.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fix: resolve PR comments
* Apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-11-24 08:16:21 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Weny Xu
dc46e96879
refactor: replace InactiveRegionManager with RegionLeaseKeeper ( #2729 )
...
* refactor: replace InactiveRegionManager with RegionLeaseKeeper
* refactor: remove InactiveRegionManager
* chore: remove remark_inactive_region field
* chore: apply suggestions from CR
* refactor: reduce meta server cache scope
* chore: remove redundant code
2023-11-14 05:16:13 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
Weny Xu
a3611516a2
refactor: heartbeat response contains region role ( #2718 )
2023-11-09 09:24:36 +00:00
Ruihang Xia
93f21b188d
chore: unify workspace dependencies and remove some unused ( #2713 )
...
* log-store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* base64
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* udeps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change the workspace deps' style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-09 07:34:22 +00:00
Weny Xu
ce867fb583
feat: introduce the region lease keeper ( #2645 )
...
* feat: introduce the region lease keeper
* chore: apply suggestions from CR
* refactor: simplify `retain_active_regions`
* refactor: remove Default of RegionStat
* chore: add todo comments
* chore: apply suggestions from CR
* refactor: simplify `retain_active_regions`
* fix: fix ci
2023-11-02 12:01:17 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Weny Xu
54ed7529ca
feat: introduce the region role ( #2640 )
...
* feat: introduce region role
* chore: apply suggestions from CR
2023-10-31 06:59:22 +00:00
Wei
36c0742c45
feat: region disk usage statistic ( #2665 )
...
* feat: region usage
* chore: code comment
* chore: unit test
* feat: region usage for engine
* chore: rename to region usage
* feat: add to heartbeat
* chore: u64 to i64 convert
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/file-engine/src/engine.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-31 03:57:37 +00:00
Niwaka
000e1471eb
feat: make mito2 have ObjectStoreManager(initial) ( #2643 )
...
* feat: make mito2 have object_store_manager(initial)
* chore: address review
* refactor: Arc<ObjectStoreManager> to ObjectStoreManagerRef and replace Vec with tuple
* fix: add ObjectStoreManager::from_default
* fix: remove cfg(test)
* fix: remove try_new from ObjectStoreManager
2023-10-30 13:16:04 +00:00
Weny Xu
dcd481e6a4
feat: stop the procedure manager if a new leader is elected ( #2576 )
...
* feat: stop the procedure manager if a new leader is elected
* chore: apply suggestions from CR
* chore: apply suggestions
* chore: apply suggestions from CR
* feat: add should_report to GreptimeDBTelemetry
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: refactor subscribing leader change loop
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2023-10-18 06:12:28 +00:00
ZhangJian He
d9e7b898a3
feat: add walconfig dir back ( #2606 )
...
Signed-off-by: ZhangJian He <shoothzj@gmail.com >
2023-10-16 11:26:06 +00:00
