WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
ff8c10eae7
feat: add CatchupRequest to engine ( #2939 )
...
* chore: remove redundant code
* feat(mito): add CatchupRequest
feat: reopen region before replay if need
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-12-21 08:27:53 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Weny Xu
97c3755ab6
feat(mito): add skip_wal_replay option to OpenRegionRequest ( #2955 )
...
* feat(mito): add skip_replay_wal option to OpenRegionRequest
* test: add tests for skip replay wal
* chore: rename `skip_replay_wal` to `skip_wal_replay`
2023-12-20 03:31:48 +00:00
niebayes
839e653e0d
feat(remote_wal): add skeleton for remote wal related to meta srv ( #2933 )
...
* feat: introduce wal config and kafka config
* feat: introduce kafka topic manager and selector
* feat: introduce region wal options
* chore: build region wal options upon starting meta srv
* feat: integrate region wal options allocator into table meta allocator
* chore: add wal config to metasrv.example.toml
* chore: add region wal options map to create table procedure
* feat: augment region create request with wal options
* feat: augment DatanodeTableValue with region wal options map
* chore: encode region wal options upon constructing table creator
* feat: persist region wal options when creating table meta
* fix: sqlness test
* chore: set default wal provider to raft-engine
* refactor: refactor wal options
* chore: update wal options allocator
* refactor: rename region wal options to wal options
* chore: update usages of region wal options
* chore: add some comments to kafka
* chore: fill in kafka config
* test: add tests for serde wal config
* test: add tests for wal options
* refactor: refactor wal options allocator to enum
* refactor: store wal options into the request options instead
* fix: typo
* fix: typo
* refactor: move wal options map to region info
* refactor: refacto serialization and deserialization of wal options
* refactor: use serde_json to encode wal options
* chore: rename wal_options_map to region_wal_options
* chore: resolve some review comments
* fix: typo
* refactor: replace kecab-case with snake_case
* fix: sqlness and converage tests
* fix: typo
* fix: coverage test
* fix: coverage test
* chore: resolve some review conversations
* fix: resolve some review conversations
* chore: format comments in metasrv.example.toml
* chore: update import style
* feat: integrate wal options allocator to standalone mode
* test: add compatible test for OpenRegion
* test: add compatible test for UpdateRegionMetadata
* chore: remove based suffix from topic selector type
2023-12-19 12:43:47 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
Zhenchi
029ff2f1e3
feat(inverted_index.create): add read/write for external intermediate files ( #2942 )
...
* feat(inverted_index.create): add read/write for external intermediate files
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: fix typos intermedia -> intermediate
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: futures_code -> asynchronous_codec
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: bump bytes to 1.5
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 09:44:48 +00:00
Ruihang Xia
9af9c0229a
feat: create table procedure for metric engine, part 1 ( #2943 )
...
* implementation
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* initialize
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove empty file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply review sugg
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-18 08:03:28 +00:00
LFC
181e16a11a
refactor: make instance started separately ( #2911 )
...
* refactor: make instance started separately, to support further integrated into other binaries
* fix: resolve PR comments
* fix: resolve PR comments
2023-12-14 03:44:29 +00:00
Yue Deng
3d651522c2
feat: add build() function to return the database build info ( #2919 )
...
* feat: add build function and register it
build() function to return the database build info #2909
* refactor: fix typos and change code structure
* test: add test for build()
* refactor: cargo fmt and eliminate warnings
* Apply suggestions from code review
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* refactor: move system.sql to a new directory
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-12-13 09:02:00 +00:00
Weny Xu
cf6bba09fd
refactor: use downgrading the region instead of closing region ( #2863 )
...
* refactor: use downgrading the region instead of closing region
* feat: enhance the tests for alive keeper
* feat: add a metric to track region lease expired
* chore: apply suggestions from CR
* chore: enable logging for test_distributed_handle_ddl_request
* refactor: simplify lease keeper
* feat: add metrics for lease keeper
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move OpeningRegionKeeper to common_meta
* feat: register operating regions to MemoryRegionKeeper
2023-12-12 09:24:17 +00:00
dennis zhuang
cf8b6c77dc
chore: clean up unused errors ( #2901 )
2023-12-11 09:08:50 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
tison
f74715ce52
refactor: RegionEngine::handle_request always returns affected rows ( #2874 )
...
* refactor: RegionEngine::handle_request -> handle_execution
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* revert spell change
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo clippy
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo fmt
Signed-off-by: tison <wander4096@gmail.com >
* more name clarification
Signed-off-by: tison <wander4096@gmail.com >
* revert rename
Signed-off-by: tison <wander4096@gmail.com >
* wrap affected rows into RegionResponse
Signed-off-by: tison <wander4096@gmail.com >
* flatten return AffectedRows
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-12-06 13:27:19 +00:00
WU Jingdi
aa89d9deef
fix: replace opendal PrometheusLayer ( #2861 )
...
* fix: replace opendal `PrometheusLayer`
* chore: add docs on `PrometheusMetricsLayer`
* chore: fix code advice
* chore: fix bug on `PrometheusMetricsLayer`
2023-12-05 07:15:45 +00:00
Weny Xu
b3ffe5cd1e
feat: handle the downgrade region instruction ( #2855 )
...
* feat: handle the downgrade region instruction
* test: add tests for RegionHeartbeatResponseHandler
* refactor: remove unused code
2023-12-05 03:30:55 +00:00
Weny Xu
7a14db68a6
feat: add upgrade candidate region step ( #2829 )
...
* feat: add upgrade candidate region step
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-12-04 05:09:27 +00:00
Weny Xu
c26f2f94c0
chore: add logs and metrics ( #2858 )
...
* chore: add logs and metrics
* feat: add the timer to track heartbeat intervel
* feat: add the gauge to track region leases
* refactor: use gauge instead of the timer
* chore: apply suggestions from CR
* feat: add hit rate and etcd txn metrics
2023-12-04 02:51:30 +00:00
Weny Xu
cce5edc88e
feat: add downgrade leader region step ( #2792 )
...
* feat: add downgrade leader region step
* chore: apply suggestions from CR
* chore: rename exist to exists
* chore: apply suggestions from CR
2023-11-29 09:17:28 +00:00
Weny Xu
9e58bba363
feat: add set_readonly_gracefully for engine ( #2787 )
...
* feat: add set_readonly_gracefully for engine
* chore: apply suggestions from CR
* chore: rename to set_readonly_test
* chore: apply suggestions from CR
2023-11-24 10:59:05 +00:00
LFC
64a36e9b36
refactor: start datanode more flexibly ( #2800 )
...
* refactor: start datanode more flexibly
* Update src/datanode/src/datanode.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fix: resolve PR comments
* Apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-11-24 08:16:21 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Weny Xu
dc46e96879
refactor: replace InactiveRegionManager with RegionLeaseKeeper ( #2729 )
...
* refactor: replace InactiveRegionManager with RegionLeaseKeeper
* refactor: remove InactiveRegionManager
* chore: remove remark_inactive_region field
* chore: apply suggestions from CR
* refactor: reduce meta server cache scope
* chore: remove redundant code
2023-11-14 05:16:13 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
Weny Xu
a3611516a2
refactor: heartbeat response contains region role ( #2718 )
2023-11-09 09:24:36 +00:00
Ruihang Xia
93f21b188d
chore: unify workspace dependencies and remove some unused ( #2713 )
...
* log-store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* base64
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* udeps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change the workspace deps' style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-09 07:34:22 +00:00
Weny Xu
ce867fb583
feat: introduce the region lease keeper ( #2645 )
...
* feat: introduce the region lease keeper
* chore: apply suggestions from CR
* refactor: simplify `retain_active_regions`
* refactor: remove Default of RegionStat
* chore: add todo comments
* chore: apply suggestions from CR
* refactor: simplify `retain_active_regions`
* fix: fix ci
2023-11-02 12:01:17 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Weny Xu
54ed7529ca
feat: introduce the region role ( #2640 )
...
* feat: introduce region role
* chore: apply suggestions from CR
2023-10-31 06:59:22 +00:00
Wei
36c0742c45
feat: region disk usage statistic ( #2665 )
...
* feat: region usage
* chore: code comment
* chore: unit test
* feat: region usage for engine
* chore: rename to region usage
* feat: add to heartbeat
* chore: u64 to i64 convert
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/file-engine/src/engine.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-31 03:57:37 +00:00
Niwaka
000e1471eb
feat: make mito2 have ObjectStoreManager(initial) ( #2643 )
...
* feat: make mito2 have object_store_manager(initial)
* chore: address review
* refactor: Arc<ObjectStoreManager> to ObjectStoreManagerRef and replace Vec with tuple
* fix: add ObjectStoreManager::from_default
* fix: remove cfg(test)
* fix: remove try_new from ObjectStoreManager
2023-10-30 13:16:04 +00:00
Weny Xu
dcd481e6a4
feat: stop the procedure manager if a new leader is elected ( #2576 )
...
* feat: stop the procedure manager if a new leader is elected
* chore: apply suggestions from CR
* chore: apply suggestions
* chore: apply suggestions from CR
* feat: add should_report to GreptimeDBTelemetry
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: refactor subscribing leader change loop
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2023-10-18 06:12:28 +00:00
ZhangJian He
d9e7b898a3
feat: add walconfig dir back ( #2606 )
...
Signed-off-by: ZhangJian He <shoothzj@gmail.com >
2023-10-16 11:26:06 +00:00
Yun Chen
f859932745
fix: convert to ReadableSize & Durations ( #2594 )
...
* fix: convert to ReadableSize & Durations
* fix: change more grpc sender/recv message size to ReadableSize
fix: format
fix: cargo fmt
fix: change cmd test to use durations
fix: revert metaclient change
fix: convert default fields in meta client options
fix: human serde meta client durations
* fix: remove milisecond postfix in heartbeat option
* fix: humantime serde on heartbeat
* fix: update config example
* fix: update integration test config
* fix: address pr comments
* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
JeremyHi
ae95f23e05
feat: add metrics for region server ( #2552 )
...
* feat: add metircs for region server
* fix: add comment and remove unused code
2023-10-10 07:40:16 +00:00
shuiyisong
007f7ba03c
refactor: extract plugins crate ( #2487 )
...
* chore: move frontend plugins fn
* chore: move datanode plugins to fn
* chore: add opt plugins
* chore: add plugins to meta-srv
* chore: setup meta plugins, wait for router extension
* chore: try use configurator for grpc too
* chore: minor fix fmt
* chore: minor fix fmt
* chore: add start meta_srv for hook
* chore: merge develop
* chore: minor fix
* chore: replace Arc<Plugins> with PluginsRef
* chore: fix header
* chore: remove empty file
* chore: modify comments
* chore: remove PluginsRef type alias
* chore: remove `OptPlugins`
2023-10-09 04:54:27 +00:00
shuiyisong
067917845f
fix: carry dbname from frontend to datanode ( #2520 )
...
* chore: add dbname in region request header for tracking purpose
* chore: fix handle read
* chore: add write meter
* chore: add meter-core to dep
* chore: add converter between RegionRequestHeader and QueryContext & update proto version
2023-10-08 06:30:23 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
Weny Xu
f50f2a84a9
fix: open region missing options ( #2473 )
...
* fix: open region missing options
* refactor: remove redundant clone
* chore: apply suggestions from CR
* chore: apply suggestions
* chore: apply suggestions
* test: add test for initialize_region_server
* feat: introduce RegionInfo
2023-10-07 07:17:16 +00:00
Zhenchi
201acd152d
fix: missing file engine with default options ( #2519 )
...
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-28 10:25:12 +00:00
Zhenchi
7edafc3407
feat: push down filters to region engine ( #2513 )
...
feat: pushdown filters to region engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-27 13:50:44 +00:00
Weny Xu
9282e59a3b
fix: re-create heartbeat stream ASAP ( #2499 )
...
* chore: set default connect_timeout_millis to 1000
* fix: re-create heartbeat stream ASAP
* chore: apply suggestions
2023-09-27 04:00:16 +00:00
Ruihang Xia
515ce825bd
feat: stack trace style debug print for error ( #2489 )
...
* impl macro stack_trace_debug
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* manually mark external error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use debug print
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify the error and warn macro
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ut
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add docs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* replace snafu backtrace with location
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-26 11:23:21 +00:00
Zhenchi
0bf26642a4
feat: re-support query engine execute dml ( #2484 )
...
* feat: re-support query engine execute dml
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: remove region_number in InsertRequest
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add doc comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-26 08:37:04 +00:00
Weny Xu
230a3026ad
fix: dn doesn't have chance to send a heartbeat to the new leader ( #2471 )
...
* refactor: set meta leader lease secs to 3s
* fix: correct default heartbeat interval
* refactor: ask meta leader in parallel
* feat: configure heartbeat client timeout to 500ms
* fix: trigger to send heartbeat immediately after fail
* fix: fix clippy
2023-09-26 05:05:38 +00:00
LinFeng
621c6f371b
feat: limit grpc message size ( #2459 )
...
* feat: add two grpc config options
Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size
* test: add integration tests for message size limit
2023-09-22 02:07:46 +00:00
Ruihang Xia
4c7ad44605
refactor: remove SqlStatementExecutor ( #2464 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-22 01:57:48 +00:00
