JohnsonLee
a50025269f
feat: Support automatic DNS lookup for kafka bootstrap servers ( #3379 )
...
* feat: Support automatic DNS lookup for kafka bootstrap servers
* Revert "feat: Support automatic DNS lookup for kafka bootstrap servers"
This reverts commit 5baed7b01dniebayes@gmail.com >
* move more common code
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
Co-authored-by: niebayes <niebayes@gmail.com >
2024-02-29 07:29:20 +00:00
SteveLauC
e9a2b0a9ee
chore: use workspace-wide lints ( #3352 )
...
* chore: use workspace-wide lints
* respond to review
2024-02-22 01:01:10 +00:00
tison
3dfe4a2e5a
chore: check dirs before create RaftEngine store ( #3327 )
...
* chore: check dirs before create RaftEngine store
Signed-off-by: tison <wander4096@gmail.com >
* fix impl
Signed-off-by: tison <wander4096@gmail.com >
* improve naming
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-02-20 07:48:15 +00:00
tison
4e04a4e48f
build: support build without git ( #3309 )
...
* build: support build without git
Signed-off-by: tison <wander4096@gmail.com >
* chore
Signed-off-by: tison <wander4096@gmail.com >
* address comment
Signed-off-by: tison <wander4096@gmail.com >
* fix syntax
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-02-18 10:30:01 +00:00
niebayes
63205907fb
refactor: introduce common-wal to aggregate wal stuff ( #3171 )
...
* refactor: aggregate wal configs
* refactor: move wal options to common-wal
* chore: slim Cargo.toml
* fix: add missing crates
* fix: format
* chore: update comments
* chore: add testing feature gate for test_util
* fix: apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix: apply suggestions from code review
* fix: compiling
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-18 03:49:37 +00:00
Weny Xu
430ffe0e28
fix(kafka): overwrite the EntryId with Offset while consuming records ( #3148 )
...
* fix(kafka): overwrite the EntryId with Offset while consuming the KafkaRecords
* fix: temporarily workaround of incorrect entry Id
2024-01-12 03:46:17 +00:00
niebayes
a0a31c8acc
chore(remote_wal): remove topic alias ( #3120 )
...
chore: remove topic alias
2024-01-09 07:35:02 +00:00
Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
niebayes
8c58d3f85b
test(remote_wal): add unit tests for kafka remote wal ( #2993 )
...
* test: add unit tests
* feat: introduce kafka runtime backed by testcontainers
* test: add test for kafka runtime
* fix: format
* chore: make kafka image ready to be used
* feat: add entry builder
* tmp
* test: add unit tests for client manager
* test: add some unit tests for kafka log store
* chore: resolve some todos
* chore: resolve some todos
* test: add unit tests for kafka log store
* chore: add deprecate develop branch warning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tmp: ready to move unit tests to an indie dir
* test: update unit tests for client manager
* test: add unit tests for meta srv remote wal
* fix: license
* fix: test
* refactor: kafka image
* doc: add doc example for kafka image
* chore: migrate kafka image to an indie PR
* fix: CR
* fix: CR
* fix: test
* fix: CR
* fix: update Cargo.toml
* fix: CR
* feat: skip test if no endpoints env
* fix: format
* test: rewrite parallel test with barrier
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-01-08 10:48:11 +00:00
Weny Xu
58ada1dfef
fix: check env before running kafka test ( #3110 )
...
* fix: check env before running kafka test
* Apply suggestions from code review
Co-authored-by: niebayes <niebayes@gmail.com >
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2024-01-08 06:30:43 +00:00
niebayes
78303639db
feat(remote_wal): split an entry if it's too large ( #3092 )
...
* feat: split an entry if it's too large
* chore: rewrite check records
* test: add some unit tests for record
* chore: rewrite entry splitting
* chore: add unit tests for build records
* chore: add more unit tests for record
* chore: rewrite encdec of record
* revert: ignored test
* fix: set limit for max_batch_size
* fix: clippy
* chore: remove heavy logging
* fix: CR
* fix: properly terminate
* fix: CR
* fix: compiling
* fix: sqlness
* fix: CR
* fix: license
* fix: license
2024-01-05 12:41:43 +00:00
Weny Xu
5653389063
feat!: correct the kafka config option ( #3065 )
...
* feat: correct the kafka config option
* refactor: rewrite the verbose comments
2024-01-02 07:31:37 +00:00
niebayes
9db168875c
fix(remote_wal): some known issues ( #3052 )
...
* fix: some known issues
* fix: CR
* fix: CR
* chore: replace Mutex with RwLock
2023-12-30 15:28:10 +00:00
Weny Xu
e16f093282
test(remote_wal): add sqlness with kafka wal ( #3027 )
...
* feat(sqlness): add kafka wal config
* chore: add sqlness with kafka wal ci config
* fix: fix config
* chore: apply suggestions from CR
* fix: add metasrv config to sqlness with kafka
* fix: replay memtable should from flushed_entry_id + 1
* fix: should set append flag to fopen
* feat: start wal allocator in standalone meta mode
* feat: append a noop record after kafka topic initialization
* test: ignore tests temporally
* test: change sqlness kafka wal config
2023-12-29 08:17:22 +00:00
Weny Xu
301ffc1d91
feat(remote_wal): append a noop record after kafka topic initialization ( #3040 )
...
* feat: append a noop record after kafka topic initialization
* chore: apply suggestions from CR
* feat: ignore the noop record during the read
2023-12-29 07:46:48 +00:00
niebayes
d061bf3d07
feat(remote_wal): introduce kafka remote wal ( #3001 )
...
* feat: integrate remote wal to standalone
* fix: test
* chore: ready to debug kafka remote wal
* fix: test
* chore: add some logs for remote wal
* chore: add logs for topic manager
* fix: properly terminate stream consumer
* fix: properly handle TopicAlreadyExists error
* fix: parse config file error
* fix: properly handle last entry id
* chore: prepare for merge
* fix: test
* fix: typo
* fix: set replication_factor properly
* fix: CR
* test: tmp for test
* Revert "test: tmp for test"
This reverts commit 093a3e0038
2023-12-26 12:35:24 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
niebayes
bab198ae68
feat(remote_wal): impl kafka log store ( #2971 )
...
* feat: introduce client manager
* chore: add errors for client manager
* chore: add record utils
* chore: impl kafka log store
* chore: build kafka log store upon starting datanode
* chore: update comments for kafka log store
* chore: add a todo for getting entry offset
* fix: typo
* chore: remove unused
* chore: update comments
* fix: typo
* fix: resolve some review conversations
* chore: move commonly referenced crates to workspace Cargo.toml
* fix: style
* fix: style
* chore: unify topic name prefix
* chore: make backoff config configurable by users
* chore: properly use backoff config in wal config
* refactor: read/write of kafka log store
* fix: typo
* fix: typo
* fix: resolve review conversations
2023-12-25 09:21:52 +00:00
niebayes
d4ac8734bc
refactor(remote_wal): entry id usage ( #2986 )
...
* chore: update comments for log store stuff
* refactor: entry id usage
* tmp update
* Revert "tmp update"
This reverts commit fcfcda2851
2023-12-25 07:30:27 +00:00
niebayes
7d509e97f6
chore: move some commonly referenced crates to workspace Cargo.toml ( #2981 )
...
fix: resolve conflicts
2023-12-22 09:13:18 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Zhenchi
029ff2f1e3
feat(inverted_index.create): add read/write for external intermediate files ( #2942 )
...
* feat(inverted_index.create): add read/write for external intermediate files
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: fix typos intermedia -> intermediate
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: futures_code -> asynchronous_codec
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: bump bytes to 1.5
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 09:44:48 +00:00
Ruihang Xia
93f21b188d
chore: unify workspace dependencies and remove some unused ( #2713 )
...
* log-store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* base64
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* udeps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change the workspace deps' style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-09 07:34:22 +00:00
Weny Xu
bd177b8cc4
refactor: remove move_value ( #2661 )
...
* chore: bump orc-rust to 0.2.42
* refactor: remove move_value
2023-10-27 03:42:50 +00:00
Weny Xu
e1dcf83326
fix: correct the range behavior in MemoryKvBackend & RaftEngineBackend ( #2615 )
...
* fix: correct the range behavior in MemoryKvBackend & RaftEngineBackend
* refactor: migrate tests from MemoryKvBackend
* chore: apply suggestions from CR
* fix: fix license header
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* fix: fix range bugs
2023-10-20 02:30:47 +00:00
Ruihang Xia
515ce825bd
feat: stack trace style debug print for error ( #2489 )
...
* impl macro stack_trace_debug
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* manually mark external error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use debug print
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify the error and warn macro
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ut
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add docs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* replace snafu backtrace with location
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-26 11:23:21 +00:00
shuiyisong
20f4f7971a
refactor: remove source and location in snafu display ( #2428 )
...
* refactor: remove source pt 1
* refactor: remove source pt 2
* refactor: remove source pt 3
* refactor: remove location pt 1
* refactor: remove location pt 2
* chore: remove rustc files
* chore: fix error case
* chore: fix test
* chore: fix test
* chore: fix cr issue
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
---------
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
2023-09-21 02:55:24 +00:00
Weny Xu
f386329e29
refactor: introduce DdlTaskExecutor and refactor statement executor ( #2341 )
...
* feat: add kv store option
* refactor: refactor statement executor
* refactor: refactor standalone table creator
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor
* fix: fix RegionDistribution
* feat: build standalone
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Weny Xu
e4de63625f
refactor: refactor raft engine backend and state store ( #2336 )
...
* refactor: remove redundant code
* refactor: refactor RaftEngineBackend Error to common_meta::error::Error
* refactor: refactor state store
* chore: apply suggestions from CR
2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1
feat: start datanode with config ( #2312 )
...
* remove memory-catalog and procedure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* derive serde for MitoConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with configs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove dir in WalConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add rename field attr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add stupid duplicated mito config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove wrong import
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wired compile error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
Lei, HUANG
9691d19601
feat: impl kv backend for raft engine ( #2280 )
...
* feat: kv backend on raft-engine
* feat: raft-engine kvbackend
* fix: toml
* fix: some review comments
* chore: optimize delete
* fix: lift lock in batch_delete
2023-09-12 07:57:15 -05:00
谢政
b62e643e92
build: update protobuf-build to support apple silicon ( #2143 )
...
* build: update protobuf-build to support apple silicon
* build: Update src/log-store/Cargo.toml
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* build: update the Cargo.lock too
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-12 03:31:51 +00:00
LFC
5c19913a91
build: on windows ( #2054 )
...
* build on windows
* rebase develop
* fix: resolve PR comments
2023-08-10 08:08:37 +00:00
Ning Sun
c39de9072f
refactor: use workspace dependencies for internal modules ( #2119 )
...
* refactor: use workspace dependencies for internal modules
* fix: resolve issue with mock module in datanode
* refactor: update test modules
2023-08-08 11:02:34 +00:00
Lei, HUANG
2d9ea595cb
chore!: change logstore namespace prefix ( #1998 )
...
* chore: change logstore namespace prefix
* chore: change delimiter
2023-08-08 07:36:46 +00:00
LFC
46fa3eb629
chore: upgrade rust toolchain to latest nightly ( #2049 )
...
* chore: upgrade rust toolchain to latest nightly
* rebase develop
* update rust toolchain in ci
2023-08-08 07:17:51 +00:00
shuiyisong
7efff2d704
fix: introduce taplo.toml and sort Cargo.toml ( #2096 )
...
* fix: add taplo.toml
* fix: introduce taplo.toml & sort cargo.toml
* chore: remove option in ci too
2023-08-04 06:44:45 +00:00
gongzhengyang
0eac56a442
chore: remove unused dependencies ( #2055 )
...
Co-authored-by: gongzhengyang <gongzhengyang@bolean.com.cn >
2023-08-01 07:43:03 +00:00
Lei, HUANG
97cfa3d6c9
feat: support append entries from multiple regions at a time ( #1959 )
...
* feat: support append entries from multiple regions at a time
* chore: add some tests
* fix: false postive mutable_key warning
* fix: append_batch api
* fix: remove unused clippy allows
2023-07-14 09:57:17 +00:00
Ben Baoyi
9ff7670adf
refactor:remove common_error::prelude ( #1946 )
...
* feat:Remove common_error::prelude
* fix merge error
* cr comment
* fix error
2023-07-13 10:36:36 +08:00
Lei, HUANG
f287d3115b
chore: replace result assertions ( #1840 )
...
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g
* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
2023-06-27 19:14:48 +08:00
LFC
fcff66e039
chore: deny unused results ( #1825 )
...
* chore: deny unused results
* rebase
2023-06-27 15:33:53 +08:00
Ruihang Xia
8c78368374
refactor: replace #[snafu(backtrace)] with Location ( #1753 )
...
* remove snafu backtrace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-06-12 11:55:33 +08:00
Lei, HUANG
82f2b34f4d
fix: wal replay ignore manifest entries ( #1612 )
...
* fix: wal replay ignore manifest entries
* test: add ut
2023-05-19 18:12:44 +08:00
Vanish
8764ce7845
feat: add delete WAL in drop_region ( #1577 )
...
* feat: add delete WAL in drop_region
* chore: fix typo err.
* feat: mark all SSTs deleted and remove the region from StorageEngine's region map.
* test: add test_drop_region for StorageEngine.
* chore: make clippy happy
* fix: fix conflict
* chore: CR.
* chore: CR
* chore: fix clippy
* fix: temp file life time
2023-05-18 18:02:34 +08:00
Lei, HUANG
e8c2222a76
feat: add WindowedReader ( #1532 )
...
* feat: add WindowedReader
* fix: some cr comments
* feat: filter memtable by timestamp range
* fix: add source in error variants
* fix: some CR comments
* refactor: filter memtable in MapIterWrapper
* fix: clippy
2023-05-17 17:34:29 +08:00
Yingwen
1833e487a4
refactor: remove unnecessary async from RepeatedTask::start ( #1545 )
...
* refactor: relax RepeatedTask requirements
Some refactor:
- Remove async from start()
- Cancel task in drop
- Allow TaskFunction::call taking &mut self
- Make start/stop concurrent safe
* test(log-store): Fix log store tests (start multiple times)
2023-05-09 21:03:15 +08:00
Ruihang Xia
da66138e80
refactor(error): remove backtrace, and introduce call-site location for debugging ( #1329 )
...
* wip: global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unneeded tests of errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ErrorExt trait implementator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-04-06 04:06:00 +00:00
dennis zhuang
563adbabe9
feat!: improve region manifest service ( #1268 )
...
* feat: try to use batch delete in ManifestLogStorage
* feat: clean temp dir when startup with file backend
* refactor: export region manifest checkpoint actions magin and refactor storage options
* feat: purge unused manifest and checkpoint files by repeat gc task
* chore: debug deleted logs
* feat: adds RepeatedTask and refactor all gc tasks
* chore: clean code
* feat: export gc_duration to manifest config
* test: assert gc works
* fix: typo
* Update src/common/runtime/src/error.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* fix: format
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* chore: by CR comments
* fix: serde default for StorageConfig
* chore: remove compaction config in StandaloneOptions
---------
Co-authored-by: LFC <bayinamine@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-03-31 10:42:00 +08:00
