niebayes
d4ac8734bc
refactor(remote_wal): entry id usage ( #2986 )
...
* chore: update comments for log store stuff
* refactor: entry id usage
* tmp update
* Revert "tmp update"
This reverts commit fcfcda2851
2023-12-25 07:30:27 +00:00
niebayes
7d509e97f6
chore: move some commonly referenced crates to workspace Cargo.toml ( #2981 )
...
fix: resolve conflicts
2023-12-22 09:13:18 +00:00
Wei
a7349b573b
feat: support fetch ranges in concurrent ( #2959 )
...
* feat: concurrent fetch ranges
* chore: cr comment
* chore: cr comment
* chore: clippy
2023-12-22 08:47:55 +00:00
Weny Xu
ff8c10eae7
feat: add CatchupRequest to engine ( #2939 )
...
* chore: remove redundant code
* feat(mito): add CatchupRequest
feat: reopen region before replay if need
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-12-21 08:27:53 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
Weny Xu
97c3755ab6
feat(mito): add skip_wal_replay option to OpenRegionRequest ( #2955 )
...
* feat(mito): add skip_replay_wal option to OpenRegionRequest
* test: add tests for skip replay wal
* chore: rename `skip_replay_wal` to `skip_wal_replay`
2023-12-20 03:31:48 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
Weny Xu
29fc2ea9d8
feat: fetch manifests in concurrent ( #2951 )
...
* feat: fetch manifests in concurrent
* chore: set fetching manifest concurrency limit to 16
2023-12-19 06:12:30 +00:00
Zhenchi
d339191e29
refactor: remove reduntant .compat() ( #2949 )
...
* refactor: remove reduntant `.compat()`
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* remove dep
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 10:36:45 +00:00
Zhenchi
029ff2f1e3
feat(inverted_index.create): add read/write for external intermediate files ( #2942 )
...
* feat(inverted_index.create): add read/write for external intermediate files
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: polish comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: fix typos intermedia -> intermediate
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: futures_code -> asynchronous_codec
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: bump bytes to 1.5
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-18 09:44:48 +00:00
Ruihang Xia
8a74bd36f5
style: rename *Adaptor to *Adapter ( #2914 )
...
* rename RecordBatchStreamAdaptor to RecordBatchStreamWrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename ServerSqlQueryHandlerAdaptor to ServerSqlQueryHandlerAdapter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-12 09:45:09 +00:00
Weny Xu
cf6bba09fd
refactor: use downgrading the region instead of closing region ( #2863 )
...
* refactor: use downgrading the region instead of closing region
* feat: enhance the tests for alive keeper
* feat: add a metric to track region lease expired
* chore: apply suggestions from CR
* chore: enable logging for test_distributed_handle_ddl_request
* refactor: simplify lease keeper
* feat: add metrics for lease keeper
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move OpeningRegionKeeper to common_meta
* feat: register operating regions to MemoryRegionKeeper
2023-12-12 09:24:17 +00:00
dennis zhuang
cf8b6c77dc
chore: clean up unused errors ( #2901 )
2023-12-11 09:08:50 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
tison
f74715ce52
refactor: RegionEngine::handle_request always returns affected rows ( #2874 )
...
* refactor: RegionEngine::handle_request -> handle_execution
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* revert spell change
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo clippy
Signed-off-by: tison <wander4096@gmail.com >
* propagate refactor
Signed-off-by: tison <wander4096@gmail.com >
* cargo fmt
Signed-off-by: tison <wander4096@gmail.com >
* more name clarification
Signed-off-by: tison <wander4096@gmail.com >
* revert rename
Signed-off-by: tison <wander4096@gmail.com >
* wrap affected rows into RegionResponse
Signed-off-by: tison <wander4096@gmail.com >
* flatten return AffectedRows
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-12-06 13:27:19 +00:00
Weny Xu
b3ffe5cd1e
feat: handle the downgrade region instruction ( #2855 )
...
* feat: handle the downgrade region instruction
* test: add tests for RegionHeartbeatResponseHandler
* refactor: remove unused code
2023-12-05 03:30:55 +00:00
Zhenchi
58c13739f0
feat(inverted_index.search): add fst applier ( #2851 )
...
* feat(inverted_index.search): add fst applier
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: typos
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-12-04 09:21:09 +00:00
Yingwen
7e68ecc498
feat: do not concat batches in MergeReader ( #2833 )
2023-12-01 06:52:43 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
Yingwen
6100cb335a
fix(mito): do not check nullability of fields in delete requests ( #2815 )
...
* test: test for delete rows from table with non null columns
* test: test delete and reopen
* fix: allow deleting rows with non null column
2023-11-27 09:54:50 +00:00
Weny Xu
9e58bba363
feat: add set_readonly_gracefully for engine ( #2787 )
...
* feat: add set_readonly_gracefully for engine
* chore: apply suggestions from CR
* chore: rename to set_readonly_test
* chore: apply suggestions from CR
2023-11-24 10:59:05 +00:00
Wei
3a4c9f2b45
feat: supports decimal type in RPC ( #2788 )
...
* refactor: ColumnDataTypeWrapper
* feat: decimal128 grpc
* feat: add test case
* chore: add TODO
* chore: empty line
* chore: remove precision and scale
* refactor: remove precision and scale
* chore: remove sqlness test
* chore: rename
* chore: proto version
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/mito2/src/memtable/time_series.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-24 09:19:33 +00:00
Yingwen
00e4bd45f0
feat: add put_only field to skip filtering deletion ( #2801 )
...
* feat: add put_only field to skip filtering deletion
* docs: fix typo
2023-11-24 06:33:17 +00:00
Ruihang Xia
99dbb7401c
refactor: remove sequence number from ScanRequest ( #2785 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-21 07:38:30 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f
feat(mito): implements row group level page cache ( #2688 )
...
* feat: add page cache
* docs: update mito config toml
* feat: impl CachedPageReader
* feat: use cache reader to read row group
* feat: do not fetch data if we have pages in cache
* chore: return if nothing to fetch
* feat: enlarge page cache size
* test: test write read parquet
* test: test cache
* docs: update comments
* test: fix config api test
* feat: cache metrics
* feat: change default page cache size
* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Ruihang Xia
ac4b6cd7f0
feat: write logical region to metric engine ( #2759 )
...
* transform write request
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add tests for put request
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use table_id instead of metric_name
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* CR sugg.
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* define random state as const
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-17 07:44:11 +00:00
Yingwen
2aa6ac5731
fix: correct memtable cost ( #2762 )
2023-11-17 02:58:51 +00:00
Wei
ae27fbc7f2
feat: impl Vector, DataType traits for decimal128 ( #2727 )
...
* feat: decimal128 vector
* feat: Decimal128Type in datatypes
* refactor: cast function
* feat: decimal128 type in ConcreteDataType
* chore: decimal
* chore: sql_data_type_to_concrete_data_type
* chore: cr comment
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: fix finish()
* chore: add debug_assert to new decimal128
---------
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-16 06:44:26 +00:00
Lei, HUANG
a24f8c96b3
perf: disable dictionary of ts columns ( #2749 )
...
* perf: disable dictionary of ts columns
* fix: unit test
2023-11-15 06:14:28 +00:00
Yingwen
f92b55c745
feat: Ignore file not found error temporarily while compacting files ( #2745 )
...
* feat: support ignoring file not found error
* feat: ignore not found during compaction
2023-11-15 03:48:51 +00:00
Ruihang Xia
e2a770f8de
fix: handle if num_cpus / 2 == 0 ( #2740 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-14 06:44:10 +00:00
Ruihang Xia
6599bb5a46
feat: do not require worker's number to be power of 2 ( #2732 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-13 09:04:44 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
Ruihang Xia
93f21b188d
chore: unify workspace dependencies and remove some unused ( #2713 )
...
* log-store
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* base64
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* udeps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* snafu style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change the workspace deps' style
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-09 07:34:22 +00:00
Yingwen
1f57c6b1f0
feat(mito): Add metrics to read path ( #2701 )
...
* feat: SST metrics and debug log
* feat: add parquet metrics
* feat: iter memtable metrics
* feat: memtable metrics and read elapsed
* feat: merge metrics
* feat: seq scan metrics
* chore: typo
* test: fix merge test
* feat: fix compiler errors
* feat: scan region log level
* feat: add build cost to seq scan metrics
* feat: adjust memtable log level
* fix: correct merge metrics
2023-11-08 07:38:04 +00:00
Lei, HUANG
c79bb5a936
fix: memtable prune ( #2698 )
...
* fix: memtable prune
* test: add memtable prune test case with complex filter
* fix: test format
2023-11-07 06:44:21 +00:00
Niwaka
e9f7579091
feat: support region ddl for custom_storage ( #2679 )
...
* feat: support region ddl for custom_storage
* fix: typo
* fix: propagate error
* refactor: have manifest_options accept RegionOptions
* chore: improve document
2023-11-06 11:18:47 +00:00
Yingwen
395632c874
feat(mito): filters memtables by their time ranges ( #2686 )
...
* feat: filter memtable by time range
* fix: incorrect time range returned by time series memtable
* test: test memtable pruning
2023-11-03 08:48:21 +00:00
Ruihang Xia
0dca63bc7b
feat: Metric engine skeleton ( #2687 )
...
* metadata region logic
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix mito2 test feature gate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* override unused warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add basic test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* apply review sugg
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy lints
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-03 08:34:08 +00:00
Yingwen
5f3bbdca4f
feat(mito): Reads SST's row groups one by one ( #2668 )
...
* feat: read parquet metadata
* feat: add create method for row group
* feat: read parquet by row group
* refactor: use VecDeque to collect batches
* style: fix row group clippy warnings
* chore: update comments
* style: fix clippy
* refactor: simplify row group reader builder
* docs: fix grammar issue
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
* chore: format code
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2023-11-01 03:59:16 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Weny Xu
54ed7529ca
feat: introduce the region role ( #2640 )
...
* feat: introduce region role
* chore: apply suggestions from CR
2023-10-31 06:59:22 +00:00
Yingwen
465c8f714e
feat(mito): avoid buffering all batches for the same primary key ( #2658 )
...
* feat: Control merge reader by batch size
* test: test heap have large range
* fix: merge one batch
* test: merge many duplicates
* test: test reheap hot
* feat: don't handle empty batch in merge reader
2023-10-31 06:36:31 +00:00
Wei
36c0742c45
feat: region disk usage statistic ( #2665 )
...
* feat: region usage
* chore: code comment
* chore: unit test
* feat: region usage for engine
* chore: rename to region usage
* feat: add to heartbeat
* chore: u64 to i64 convert
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/file-engine/src/engine.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-31 03:57:37 +00:00
Niwaka
000e1471eb
feat: make mito2 have ObjectStoreManager(initial) ( #2643 )
...
* feat: make mito2 have object_store_manager(initial)
* chore: address review
* refactor: Arc<ObjectStoreManager> to ObjectStoreManagerRef and replace Vec with tuple
* fix: add ObjectStoreManager::from_default
* fix: remove cfg(test)
* fix: remove try_new from ObjectStoreManager
2023-10-30 13:16:04 +00:00
Wei
958ff3f185
feat: estimate region wal size ( #2652 )
...
* feat: estimated wal size
* chore: modify factor
* chore: cr comment
* chore: add TODO
2023-10-27 02:46:24 +00:00
Ruihang Xia
5d8b0e8154
chore: bump dependencies ( #2659 )
...
* fix all hard error
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix nextest
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* trivial changes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix order by
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix sql keyword and data type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix range exec's input partitioning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix cover input type
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix explain analyze
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update distributed mode sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix lints
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update locks
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* downgrade dlv-list
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-10-26 12:24:14 +00:00
Yingwen
16a3257ada
fix: add serde default to MitoConfig ( #2650 )
...
* fix: add serde default to MitoConfig
* docs: update config toml
2023-10-24 11:24:47 +00:00
