Zhenchi
04ac0c8da0
feat(fulltext_index): integrate full-text indexer with parquet reader ( #4348 )
...
* feat(fulltext_index): integrate full-text indexer with parquet reader
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* disable reload
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: range allow exceeding total row
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: unit tests in index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: prune row groups
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: rename creator
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: sst fulltext index
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: address comment
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-07-15 08:14:44 +00:00
LFC
f4a5a44549
refactor: make region manifest checkpoint ran in background ( #4133 )
...
* refactor: use in-database manifest as checkpoint instead of merging incremental files in object store
* refactor: make region manifest checkpoint ran in background
* reduce unnecessary metrics
* Update src/mito2/src/manifest/checkpointer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* resolve PR comments
* resolve PR comments
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-17 03:47:18 +00:00
Weny Xu
a6269397c8
fix: fix EntityTooSmall issue ( #4100 )
...
* fix: fix EntityTooSmall issue
* chore(ci): add minio to coverage
* tests: add test for parquet writer
* chore: move tests to `common-datasource` crate
2024-06-04 08:43:33 +00:00
Weny Xu
b3a4362626
test: run test_flush_reopen_region and test_region_replay with KafkaLogStore ( #4083 )
...
* feat: add `LogStoreFactory` to `TestEnv`
* feat: add `multiple_log_store_factories` template
* test: run `test_flush_reopen_region` and `test_region_replay` with `KafkaLogStore`
* chore: move deps to workspace
* chore: apply suggestions from CR
2024-06-04 08:11:15 +00:00
Yingwen
4685b59ef1
feat: write manifests in background tasks ( #3709 )
...
* chore: truncate wip
* feat: truncate and edit write manifest in background
* refactor: wrap in manifest context
* feat: alter write manifest in background
* chore: fix compiler errors
* feat: flush update manifest in background
* feat: compaction update manifest in background
* feat: set dropping state
* feat: reset drop state
* feat: check state before updating manifest
* test: fix compaction test
* refactor: rename method
* chore: update comment
* chore: discard state guard
* refactor: use atomic cell to store state enum
* chore: fix clippy
* chore: update toml
* chore: remove unused type alias
* feat: check state after writing manifest
* chore: address CR comments
* chore: change status code
* chore: Update src/mito2/src/region.rs
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* fix: executes applier
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2024-04-24 03:09:48 +00:00
tison
2c0c7759ee
feat: add checksum for checkpoint data ( #3651 )
...
* feat: add checksum for checkpoint data
Signed-off-by: tison <wander4096@gmail.com >
* add test
Signed-off-by: tison <wander4096@gmail.com >
* clippy
Signed-off-by: tison <wander4096@gmail.com >
* fix: checksum should calculate on uncompressed data
Signed-off-by: tison <wander4096@gmail.com >
* address comments
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-04-09 08:32:24 +00:00
Yingwen
ffbb132f27
feat: Implement an unordered scanner for append mode ( #3598 )
...
* feat: ScanInput
* refactor: seq scan use scan input
* chore: implement unordered scan
* feat: use unordered scan for append table
* fix: unordered scan panic
* docs: update mermaid
* chore: address comment
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2024-03-29 07:25:35 +00:00
dimbtp
dd18d8c97b
build(deps): remove some unused dependencies ( #3582 )
...
* build(deps): remove some unused dependencies
* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
Yingwen
641592644d
feat: support per table memtable options ( #3524 )
...
* feat: add memtable builder to region
* refactor: rename memtable_builder in worker to default_memtable_builder
* fix: return error instead of using default compaction options
Support deserializing memtable and compaction options from the option
map
* feat: optional memtable options
* feat: add MemtableBuilderProvider to create builders
* feat: change default memtable and skip deserializing dedup
* chore: update test and comment
* chore: test invalid type
* feat: metric engine use new memtable manually
* feat: expose more memtable configs
* feat: add memtable options to valid option list
* test: add test
* test: sqlness test
* chore: serde workspace
* chore: remove comments
2024-03-19 08:50:10 +00:00
Yingwen
74862f8c3f
feat(mito): Checks whether a region should flush periodically ( #3459 )
...
* feat: handle flush periodically
* chore: call periodical method in loop
* feat: check periodical tasks on channel timeout
* refactor: use time provider to get time
Mock a time provider to test auto flush
* chore: fix typos
* refactor: rename mock time provider
* style: fix cilppy
* chore: address comment
2024-03-15 06:41:28 +00:00
Yingwen
7c895e2605
perf: more benchmarks for memtables ( #3491 )
...
* chore: remove duplicate bench
* refactor: rename bench
* perf: add full scan bench for memtable
* feat: filter bench and add time series to bench group
* chore: comment
* refactor: rename
* style: fix clippy
2024-03-12 12:02:58 +00:00
Lei, HUANG
7183fa198c
refactor: make MergeTreeMemtable the default choice ( #3430 )
...
* refactor: make MergeTreeMemtable the default choice
* refactor: reformat
* chore: add doc to config
2024-03-05 10:00:08 +00:00
Lei, HUANG
376409b857
feat: employ sparse key encoding for shard lookup ( #3410 )
...
* feat: employ short key encoding for shard lookup
* fix: license
* chore: simplify code
* refactor: only enable sparse encoding to speed lookup on metric engine
* fix: names
2024-03-01 06:22:15 +00:00
SteveLauC
e9a2b0a9ee
chore: use workspace-wide lints ( #3352 )
...
* chore: use workspace-wide lints
* respond to review
2024-02-22 01:01:10 +00:00
Yingwen
f087a843bb
feat: Implement KeyDictBuilder for the merge tree memtable ( #3334 )
...
* feat: dict builder
* feat: write and scan dict builder
* chore: address CR comments
2024-02-20 15:39:17 +00:00
tison
4e04a4e48f
build: support build without git ( #3309 )
...
* build: support build without git
Signed-off-by: tison <wander4096@gmail.com >
* chore
Signed-off-by: tison <wander4096@gmail.com >
* address comment
Signed-off-by: tison <wander4096@gmail.com >
* fix syntax
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-02-18 10:30:01 +00:00
Zhenchi
141ed51dcc
feat(mito): adjust seg size of inverted index to finer granularity instead of row group level ( #3289 )
...
* feat(mito): adjust seg size of inverted index to finer granularity instead of row group level
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: wrong metric
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: more suitable name
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: BitVec instead
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-02-07 08:20:00 +00:00
niebayes
63205907fb
refactor: introduce common-wal to aggregate wal stuff ( #3171 )
...
* refactor: aggregate wal configs
* refactor: move wal options to common-wal
* chore: slim Cargo.toml
* fix: add missing crates
* fix: format
* chore: update comments
* chore: add testing feature gate for test_util
* fix: apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* fix: apply suggestions from code review
* fix: compiling
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-18 03:49:37 +00:00
Wei
3d7d2fdb4a
feat: auto config cache size according to memory size ( #3165 )
...
* feat: auto config cache and buffer size according to mem size
* feat: utils
* refactor: add util function to common config
* refactor: check cgroups
* refactor: code
* fix: test
* fix: test
* chore: cr comment
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* chore: remove default comment
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
2024-01-17 14:35:35 +00:00
Zhenchi
d973cf81f0
feat(inverted_index): implement apply for SstIndexApplier ( #3088 )
...
* feat(inverted_index): implement apply for SstIndexApplier
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: rename metrics
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-01-04 07:33:03 +00:00
Yingwen
2e686fe053
feat(mito): Implement file cache ( #3022 )
...
* feat: recover cache
* feat: moka features
* test: tests for file cache
* chore: suppress warninig
* fix: parse_inde_key consider suffix
* feat: update cache
* feat: expose cache file path
* feat: use cache_path in test
2024-01-03 02:05:06 +00:00
Zhenchi
69a53130c2
feat(inverted_index): Add applier builder to convert Expr to Predicates (Part 1) ( #3034 )
...
* feat(inverted_index.integration): Add applier builder to convert Expr to Predicates (Part 1)
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: add docs
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/sst/index/applier/builder.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: remove unwrap
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: error source
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-12-30 07:32:32 +00:00
niebayes
7d509e97f6
chore: move some commonly referenced crates to workspace Cargo.toml ( #2981 )
...
fix: resolve conflicts
2023-12-22 09:13:18 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Zhenchi
d339191e29
refactor: remove reduntant .compat() ( #2949 )
...
* refactor: remove reduntant `.compat()`
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* remove dep
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 10:36:45 +00:00
Zhenchi
029ff2f1e3
feat(inverted_index.create): add read/write for external intermediate files ( #2942 )
...
* feat(inverted_index.create): add read/write for external intermediate files
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: fix typos intermedia -> intermediate
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: futures_code -> asynchronous_codec
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: bump bytes to 1.5
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 09:44:48 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Wei
ae27fbc7f2
feat: impl Vector, DataType traits for decimal128 ( #2727 )
...
* feat: decimal128 vector
* feat: Decimal128Type in datatypes
* refactor: cast function
* feat: decimal128 type in ConcreteDataType
* chore: decimal
* chore: sql_data_type_to_concrete_data_type
* chore: cr comment
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: fix finish()
* chore: add debug_assert to new decimal128
---------
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-16 06:44:26 +00:00
Ruihang Xia
6599bb5a46
feat: do not require worker's number to be power of 2 ( #2732 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-13 09:04:44 +00:00
Ruihang Xia
93f21b188d
chore: unify workspace dependencies and remove some unused ( #2713 )
...
* log-store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* base64
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* udeps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change the workspace deps' style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-09 07:34:22 +00:00
Ruihang Xia
0dca63bc7b
feat: Metric engine skeleton ( #2687 )
...
* metadata region logic
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix mito2 test feature gate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* override unused warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add basic test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply review sugg
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy lints
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-03 08:34:08 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Lei, HUANG
8bdef9a348
feat: memtable filter push down ( #2539 )
...
* feat: memtable support filter pushdown to prune primary keys
* fix: switch to next time series when pk not selected
* fix: allow predicate evaluation failure
* fix: some clippy warnings
* fix: panic when no primary key in schema
* feat: cache decoded record batch for primary key
* refactor: use arcswap instead of rwlock
* fix: format toml
2023-10-10 04:03:10 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
Yingwen
a6116bb866
feat(mito): Add cache manager ( #2488 )
...
* feat: add cache manager
* feat: add cache to reader builder
* feat: add AsyncFileReaderCache
* feat: Impl AsyncFileReaderCache
* chore: move moka dep to workspace
* feat: add moka cache to the manager
* feat: implement parquet meta cache
* test: test cache manager
* feat: consider vec size
* style: fix clippy
* test: fix config api test
* feat: divide cache
* test: test disabling meta cache
* test: fix config api test
* feat: remove meta cache if file is purged
2023-09-26 11:46:19 +00:00
Ruihang Xia
515ce825bd
feat: stack trace style debug print for error ( #2489 )
...
* impl macro stack_trace_debug
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* manually mark external error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use debug print
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* simplify the error and warn macro
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ut
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add docs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* replace snafu backtrace with location
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-26 11:23:21 +00:00
Yingwen
7b606ed289
feat(mito): make use of options in RegionCreate/OpenRequest ( #2436 )
...
* refactor: move RegionOptions to options mod
* refactor: define compaction strategy in region/options.rs
* feat: use duration for time window
* refactor: rename CompactionStrategy to CompactionOptions
* feat: use serde to parse options
* feat: parse options
* feat: set options on creation/opening
* test: test create/open with options
* chore: remove todo
* feat: get compaction ttl and options from RegionOptions
* style: fix clippy
* chore: Remove unused engine_options
* style: fix clippy
* chore: remove todo
2023-09-19 09:06:09 +00:00
Yingwen
4a82926d72
docs: fix cargo doc errors and warnings ( #2421 )
...
* docs: fix cargo doc warnings and errors
* docs: fix warnings
* docs: fix warnings
* chore: rm src/common/function-macro/src/lib.rs
2023-09-17 11:45:15 +00:00
dennis zhuang
627c5b7419
feat: move table operations from frontend to operator crate ( #2411 )
...
* feat: move table operations from frontend to operator crate
* chore: blank line
* fix: toml format
* chore: move constants
2023-09-16 07:58:45 +00:00
Lei, HUANG
3331e3158c
feat(mito2): compaction ( #2317 )
...
* feat: compaction component
* feat: mito2 compaction
* Avoid building time range predicates when merge SST files since in TWCS we don't enforce strict time window.
* fix: some CR comments
* minor: change CompactionRequest::senders to an option
* chore: handle compaction finish error
* feat: integrate compaction into region worker
* chore: rebase upstream
* fix: Some CR comments
* chore: Apply suggestions from code review
* style: fix clippy
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1
feat: start datanode with config ( #2312 )
...
* remove memory-catalog and procedure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* derive serde for MitoConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* start datanode with configs
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove dir in WalConfig
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add rename field attr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add stupid duplicated mito config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove wrong import
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wired compile error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-12 07:57:15 -05:00
Yingwen
648b2ae293
feat(mito): Flush region ( #2291 )
...
* chore: call handle_flush_request
* feat: alias SchedulerRef and clean scheduler on drop
* feat: add scheduler to workers
* feat: remove RegionMemtableStats
* feat: pick regions to flush
* feat: add more fields to region flush task
* feat: smallvec workspace dep
* feat: Use list to hold immutable memtables
* feat: flush job wip
* feat: use access layer to read write sst
* feat: flush memtables to l0
* feat: write manifest
* feat: schedule next flush on success
* feat: schedule flush on success and failure
* feat: add purger to region
* feat: apply edit after flush
* feat: collect stats for SSTs
* feat: manual flush
* test: test flush and fix manifest test
* feat: remove flush scheduler job limit
* fix: typo
* style: clippy
* feat: clean flushed files on failure
* chore: address CR comment
* refactor: Use put_rows
* feat: Clean flush scheduler on drop
* feat: remove region flush status on drop and close
* chore: address CR comment
2023-09-12 07:57:15 -05:00
Lei, HUANG
86d56f71ef
fix: flume bug ( #2298 )
...
fix: flume
2023-09-12 07:57:15 -05:00
LFC
ef75e8f7c3
feat: create distributed Mito2 table ( #2246 )
...
* feat: create distributed Mito2 table
* rebase develop
2023-08-28 12:07:52 +00:00
Bamboo1
4dbc32f532
refactor: remove associate type in scheduler to simplify it #2153 ( #2194 )
...
* feature: add a simple scheduler using flume
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: only use a sender rather clone many senders
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: use select to avoid loop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: add parameters in new function to build the flume capacity and number of receivers
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: add countdownlatch test concurrency
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: add barrier replacing countdownlatch to test concurrency and add wait all tasks finished in stop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add some document annotation
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add license header
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add Cargo.lock
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: Cargo.toml format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: delete println in test
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: add error handle
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: fix error handle and add test scheduler stop
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: spelling mistake
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: wait all tasks finished
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: add todo which need wrap Future returned by send_async
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* test: remove unnessary sleep in test
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: resolve some conflicts
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* fix: resolve conversation
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* chore: code format
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
* feat: modify the function of schedule to synchronize and drop sender after stopping scheduler
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
---------
Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com >
2023-08-23 06:28:00 +00:00
Lei, HUANG
e924b44e83
refactor: KeyValues return ValueRef ( #2170 )
...
* refactor: KeyValues return ValueRef
* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.
* feat: define min value of OpType as a const
* fix: toml format
2023-08-14 14:51:13 +00:00
Lei, HUANG
55b5df9c51
feat: row wise converter ( #2162 )
...
* feat: impl mem-comparable encoding for timestamp
* fix: test cases
* impl time series encode/decoder
* fix: merge unsupported match arms
* fix: clippy
* chore: big number delimiter
* feat: encode timestamps as i64
* fix: remove useless error variant
2023-08-14 07:13:39 +00:00
Ruihang Xia
6d64e1c296
feat(mito): checkpoint for mito2 ( #2142 )
...
* basic impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* adjust dir structure
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix styles
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* sort result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* downgrade log level
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply CR sugg.
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add region id to log
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-08-13 09:26:01 +00:00
Ning Sun
c39de9072f
refactor: use workspace dependencies for internal modules ( #2119 )
...
* refactor: use workspace dependencies for internal modules
* fix: resolve issue with mock module in datanode
* refactor: update test modules
2023-08-08 11:02:34 +00:00
Yingwen
cb4dd89754
feat(mito): Implement mito2 Wal ( #2103 )
...
* feat: define wal struct
* feat: Implement Wal read/write
* feat: obsolete wal
* test: test wal
* refactor: use try_stream and remove async from scan
2023-08-04 11:04:25 +00:00
