fys
74bfb09195
feat: support cache for batch_get in CachedMetaKvBackend ( #3277 )
...
* feat: support cache for batch_get in CachedMetaKvBackend.
* add doc of CachedMetaKvBackend
* fix: cr
* fix: correct some words
* fix cr
2024-02-06 03:15:03 +00:00
Weny Xu
19413eb345
refactor!: rename initialize_region_in_background to init_regions_in_background ( #3216 )
...
refactor!: change initialize_region_in_background to init_regions_in_background
2024-01-23 04:33:48 +00:00
Ruihang Xia
31787f4bfd
feat!: switch prom remote write to metric engine ( #3198 )
...
* feat: switch prom remote write to metric engine
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* read physical table name from url
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove physical table from header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix merge error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add with_metric_engine option to config remote write behavior
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* check parameter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add specific config param
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* default with_metric_engine to true
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update UT
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 14:48:28 +00:00
shuiyisong
2bf4b08a6b
chore: change default factor to compute memory size ( #3211 )
...
* chore: change default factor to compute memory size
* chore: update doc
* chore: update comment in example config
* chore: extract factor to const and update comments
* chore: update comment by cr suggestion
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-01-22 09:03:29 +00:00
niebayes
63205907fb
refactor: introduce common-wal to aggregate wal stuff ( #3171 )
...
* refactor: aggregate wal configs
* refactor: move wal options to common-wal
* chore: slim Cargo.toml
* fix: add missing crates
* fix: format
* chore: update comments
* chore: add testing feature gate for test_util
* fix: apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix: apply suggestions from code review
* fix: compiling
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-18 03:49:37 +00:00
Zhenchi
6f07d69155
feat(mito): enable inverted index ( #3158 )
...
* feat(mito): enable inverted index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* accidentally resolved the incorrect filtering issue within the Metric Engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/access_layer.rs
* Update src/mito2/src/test_util/scheduler_util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: format -> join_dir
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: move intermediate_manager from arg of write_and_upload_sst to field of WriteCache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* refactor: add IndexerBuidler
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix clippy
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-01-15 09:08:07 +00:00
niebayes
f5798e2833
fix: remove incorrect wal comments in config file ( #3142 )
...
fix: kafka config comments
2024-01-11 09:34:24 +00:00
niebayes
1b0e39a7f2
chore: stop exposing num_partitions ( #3132 )
2024-01-10 10:45:18 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
niebayes
78303639db
feat(remote_wal): split an entry if it's too large ( #3092 )
...
* feat: split an entry if it's too large
* chore: rewrite check records
* test: add some unit tests for record
* chore: rewrite entry splitting
* chore: add unit tests for build records
* chore: add more unit tests for record
* chore: rewrite encdec of record
* revert: ignored test
* fix: set limit for max_batch_size
* fix: clippy
* chore: remove heavy logging
* fix: CR
* fix: properly terminate
* fix: CR
* fix: compiling
* fix: sqlness
* fix: CR
* fix: license
* fix: license
2024-01-05 12:41:43 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
Weny Xu
5653389063
feat!: correct the kafka config option ( #3065 )
...
* feat: correct the kafka config option
* refactor: rewrite the verbose comments
2024-01-02 07:31:37 +00:00
zyy17
b8b1e98399
refactor: use string type instead of Option type for '--store-key-prefix' ( #3018 )
...
* refactor: use string type instead of Option type for '--store-key-prefix'
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: refine for code review comments
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com >
2023-12-27 11:26:30 +00:00
WU Jingdi
d1ee1ba56a
feat: support set timezone in db ( #2992 )
...
* feat: support set timezone in db
* chore: fix ci
* chore: fix code advice
* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
niebayes
d061bf3d07
feat(remote_wal): introduce kafka remote wal ( #3001 )
...
* feat: integrate remote wal to standalone
* fix: test
* chore: ready to debug kafka remote wal
* fix: test
* chore: add some logs for remote wal
* chore: add logs for topic manager
* fix: properly terminate stream consumer
* fix: properly handle TopicAlreadyExists error
* fix: parse config file error
* fix: properly handle last entry id
* chore: prepare for merge
* fix: test
* fix: typo
* fix: set replication_factor properly
* fix: CR
* test: tmp for test
* Revert "test: tmp for test"
This reverts commit 093a3e0038
2023-12-26 12:35:24 +00:00
LFC
196c06db14
feat: make logging to stdout configurable ( #3003 )
...
* feat: make logging to stdout configurable
* fix sqlness
* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
niebayes
bab198ae68
feat(remote_wal): impl kafka log store ( #2971 )
...
* feat: introduce client manager
* chore: add errors for client manager
* chore: add record utils
* chore: impl kafka log store
* chore: build kafka log store upon starting datanode
* chore: update comments for kafka log store
* chore: add a todo for getting entry offset
* fix: typo
* chore: remove unused
* chore: update comments
* fix: typo
* fix: resolve some review conversations
* chore: move commonly referenced crates to workspace Cargo.toml
* fix: style
* fix: style
* chore: unify topic name prefix
* chore: make backoff config configurable by users
* chore: properly use backoff config in wal config
* refactor: read/write of kafka log store
* fix: typo
* fix: typo
* fix: resolve review conversations
2023-12-25 09:21:52 +00:00
niebayes
830a91c548
feat(remote_wal): implement topic allocation ( #2970 )
...
* chore: implement wal options allocator
* chore: implement round-robin topic selector
* feat: add shuffle to round-robin topic selector
* chore: implement kafka topic manager
* test: add tests for wal options allocator
* test: add wal provider to test config files
* test: leave todos for adding tests for remote wal
* fix: resolve review conversations
* fix: typo
2023-12-22 08:26:48 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
JeremyHi
5f8c17514f
chore: SelectorType use snake_case ( #2962 )
...
chore: let SelectorType use snake_case
2023-12-20 02:28:29 +00:00
niebayes
839e653e0d
feat(remote_wal): add skeleton for remote wal related to meta srv ( #2933 )
...
* feat: introduce wal config and kafka config
* feat: introduce kafka topic manager and selector
* feat: introduce region wal options
* chore: build region wal options upon starting meta srv
* feat: integrate region wal options allocator into table meta allocator
* chore: add wal config to metasrv.example.toml
* chore: add region wal options map to create table procedure
* feat: augment region create request with wal options
* feat: augment DatanodeTableValue with region wal options map
* chore: encode region wal options upon constructing table creator
* feat: persist region wal options when creating table meta
* fix: sqlness test
* chore: set default wal provider to raft-engine
* refactor: refactor wal options
* chore: update wal options allocator
* refactor: rename region wal options to wal options
* chore: update usages of region wal options
* chore: add some comments to kafka
* chore: fill in kafka config
* test: add tests for serde wal config
* test: add tests for wal options
* refactor: refactor wal options allocator to enum
* refactor: store wal options into the request options instead
* fix: typo
* fix: typo
* refactor: move wal options map to region info
* refactor: refacto serialization and deserialization of wal options
* refactor: use serde_json to encode wal options
* chore: rename wal_options_map to region_wal_options
* chore: resolve some review comments
* fix: typo
* refactor: replace kecab-case with snake_case
* fix: sqlness and converage tests
* fix: typo
* fix: coverage test
* fix: coverage test
* chore: resolve some review conversations
* fix: resolve some review conversations
* chore: format comments in metasrv.example.toml
* chore: update import style
* feat: integrate wal options allocator to standalone mode
* test: add compatible test for OpenRegion
* test: add compatible test for UpdateRegionMetadata
* chore: remove based suffix from topic selector type
2023-12-19 12:43:47 +00:00
dennis zhuang
600cde1ff2
fix: wrong link for selector ( #2958 )
2023-12-19 10:50:48 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
09aa4b72a5
chore: update storage config example ( #2887 )
...
chore: update config example
2023-12-07 03:18:56 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
WU Jingdi
0badb3715e
feat: support sample ratio in trace ( #2809 )
...
* feat: support sample ratio in trace
* chore: fix code advice
2023-11-27 06:46:46 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f
feat(mito): implements row group level page cache ( #2688 )
...
* feat: add page cache
* docs: update mito config toml
* feat: impl CachedPageReader
* feat: use cache reader to read row group
* feat: do not fetch data if we have pages in cache
* chore: return if nothing to fetch
* feat: enlarge page cache size
* test: test write read parquet
* test: test cache
* docs: update comments
* test: fix config api test
* feat: cache metrics
* feat: change default page cache size
* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074
fix!: improve user experience on setting compression type ( #2765 )
...
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type
* Update src/common/datasource/src/compression.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-17 06:27:13 +00:00
tison
af7107565a
refactor: use humantime for user-facing options ( #2724 )
...
* refactor: use humantime for user-facing options
Signed-off-by: tison <wander4096@gmail.com >
* unify quote style
Signed-off-by: tison <wander4096@gmail.com >
* taplo fmt
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-11-09 13:29:47 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
tison
b9a7c2db7e
feat: make PhiAccrualFailureDetector configurable ( #2709 )
...
* feat: make PhiAccrualFailureDetector configurable
Signed-off-by: tison <wander4096@gmail.com >
* propagate
Signed-off-by: tison <wander4096@gmail.com >
* f32 does not implement Eq
Signed-off-by: tison <wander4096@gmail.com >
* add to test
Signed-off-by: tison <wander4096@gmail.com >
* fmt clippy
Signed-off-by: tison <wander4096@gmail.com >
* Update src/meta-srv/src/failure_detector.rs
* address comments
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-11-09 06:26:33 +00:00
Yingwen
16a3257ada
fix: add serde default to MitoConfig ( #2650 )
...
* fix: add serde default to MitoConfig
* docs: update config toml
2023-10-24 11:24:47 +00:00
ZhangJian He
d9e7b898a3
feat: add walconfig dir back ( #2606 )
...
Signed-off-by: ZhangJian He <shoothzj@gmail.com >
2023-10-16 11:26:06 +00:00
Yun Chen
f859932745
fix: convert to ReadableSize & Durations ( #2594 )
...
* fix: convert to ReadableSize & Durations
* fix: change more grpc sender/recv message size to ReadableSize
fix: format
fix: cargo fmt
fix: change cmd test to use durations
fix: revert metaclient change
fix: convert default fields in meta client options
fix: human serde meta client durations
* fix: remove milisecond postfix in heartbeat option
* fix: humantime serde on heartbeat
* fix: update config example
* fix: update integration test config
* fix: address pr comments
* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
Ruihang Xia
62bcb45787
feat!: change config name from kv_store to metadata_store ( #2585 )
...
featchange config name from kv_store to metadata_store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-12 06:55:09 +00:00
dennis zhuang
0a9972aa9a
fix: cache capacity unit in sample config ( #2575 )
2023-10-11 11:02:39 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
Weny Xu
9282e59a3b
fix: re-create heartbeat stream ASAP ( #2499 )
...
* chore: set default connect_timeout_millis to 1000
* fix: re-create heartbeat stream ASAP
* chore: apply suggestions
2023-09-27 04:00:16 +00:00
Weny Xu
230a3026ad
fix: dn doesn't have chance to send a heartbeat to the new leader ( #2471 )
...
* refactor: set meta leader lease secs to 3s
* fix: correct default heartbeat interval
* refactor: ask meta leader in parallel
* feat: configure heartbeat client timeout to 500ms
* fix: trigger to send heartbeat immediately after fail
* fix: fix clippy
2023-09-26 05:05:38 +00:00
JeremyHi
7310ec0bb3
chore: refactor options ( #2476 )
2023-09-24 02:12:33 +00:00
Weny Xu
43476e1ff9
refactor: rename coordination to require_lease_before_startup ( #2431 )
2023-09-18 11:07:42 +00:00
Weny Xu
2c3ff90dbc
feat: start services after first heartbeat response processed ( #2424 )
...
* feat: start services after first heartbeat response processed
* refactor: watch changes in RegionAliveKeeper
* feat: add coordination to DatanodeOptions
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: enable coordination in sqlness
2023-09-18 08:49:26 +00:00
JeremyHi
e7e254cd11
feat: all distributed time together ( #2423 )
2023-09-17 15:18:52 +00:00
