rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-19 14:30:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,286 Commits 133 Branches 253 Tags
feat/expr-split
Commit Graph

857 Commits

Author SHA1 Message Date
Ruihang Xia
50fca2400e feat: adapt methods from RegionEngine for MitoEngine (#2315) 
* feat: adapt methods from RegionEngine for MitoEngine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* minor fixes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Yingwen
50220f8f04 feat: Impl write buffer manager for mito2 (#2309) 
* feat: add write buffer manager to builder

* feat: impl WriteBufferManager

* feat: impl MemtableVersion::mutable_usage

* chore: Address CR comments

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* refactor: rename mutable_limitation to mutable_limit

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
dennis zhuang
fad58835bf fix: don't raise an error when manifest directory is not created (#2308) 
* fix: don't raise an error when manifest directory is not created

* chore: apply suggestion

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-09-12 07:57:15 -05:00
Lei, HUANG
43fdff3639 feat: remove memtable request (#2307) 
* refactor: remove scan request from memtable API

* docs: Update comment

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
Lei, HUANG
36231a5d50 feat(mito2): add alloc_tracker for memtable (#2266) 
* feat: add alloc_tracker for memtable

* chore: integrate WriteBufferManager
 2023-09-12 07:57:15 -05:00
Yingwen
648b2ae293 feat(mito): Flush region (#2291) 
* chore: call handle_flush_request

* feat: alias SchedulerRef and clean scheduler on drop

* feat: add scheduler to workers

* feat: remove RegionMemtableStats

* feat: pick regions to flush

* feat: add more fields to region flush task

* feat: smallvec workspace dep

* feat: Use list to hold immutable memtables

* feat: flush job wip

* feat: use access layer to read write sst

* feat: flush memtables to l0

* feat: write manifest

* feat: schedule next flush on success

* feat: schedule flush on success and failure

* feat: add purger to region

* feat: apply edit after flush

* feat: collect stats for SSTs

* feat: manual flush

* test: test flush and fix manifest test

* feat: remove flush scheduler job limit

* fix: typo

* style: clippy

* feat: clean flushed files on failure

* chore: address CR comment

* refactor: Use put_rows

* feat: Clean flush scheduler on drop

* feat: remove region flush status on drop and close

* chore: address CR comment
 2023-09-12 07:57:15 -05:00
Ruihang Xia
eddff17523 feat: drop region in mito2 (#2286) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check in opening region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/mito2/src/worker/handle_drop.rs

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* remove file in order

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remove logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use scan to list files

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
Yingwen
b234733c61 feat(mito): Support deleting rows in mito2 (#2275) 
* feat: check delete request

* test: test delete and overwrite
 2023-09-12 07:57:15 -05:00
Lei, HUANG
86d56f71ef fix: flume bug (#2298) 
fix: flume
 2023-09-12 07:57:15 -05:00
Yingwen
365e557e7a feat(mito): Integrate access layer and file purger to region (#2296) 
* feat: alias SchedulerRef and clean scheduler on drop

* feat: add scheduler to workers

* feat: use access layer to read write sst

* feat: add purger to region

* refactor: allow getting region_dir from AccessLayer

* feat: add scheduler to FlushScheduler

* feat: getter for object store

* chore: fix typo

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Bamboo1
68600a2cf9 feat(mito2): add file purger and cooperate with scheduler to purge sst files (#2251) 
* feat: add file purger and use scheduler

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: print some information about handling error message

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: resolve conversion

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: resolve conversation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: resolve conflicting files

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

---------

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>
 2023-08-29 07:55:03 +00:00
Yingwen
805f254d15 feat(mito): Flush framework for mito2 (#2262) 
* feat: write buffer manager

* feat: skeleton

* feat: add flush logic to write path

* feat: add methods to memtable trait

* feat: freeze memtable

* feat: define flush task

* feat: schedule_flush wip

* feat: adding pending requests/tasks

* feat: separate ddl request and background request

* feat: Remove RegionTask and RequestBody

* feat: handle flush related requests

* feat: make tests pass

* style: fix clippy

* docs: update comment

* refactor: rename background requests

* feat: replace Option<RegionWriteCtx> with an enum MaybeStalling
 2023-08-29 07:13:15 +00:00
LFC
ef75e8f7c3 feat: create distributed Mito2 table (#2246) 
* feat: create distributed Mito2 table

* rebase develop
 2023-08-28 12:07:52 +00:00
Lei, HUANG
c112b9a763 feat(mito2): WAL replay (#2264) 
* feat: replay memtable when opening table

* test: region replay

* refactor: save logstore in TestEnv

* fix: some cr comments

* chore: rebase develop

* chore: update last entry id during replay
 2023-08-28 11:45:23 +00:00
Ruihang Xia
6b8cf0bbf0 feat: impl region engine for mito (#2269) 
* update proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* convert request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* import result convertor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename symbols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-28 09:24:12 +00:00
Yingwen
e2522dff21 feat(mito): Skeleton for scanning a region (#2230) 
* feat: define stream builder

* feat: scan region wip

* feat: create SeqScan in ScanRegion

* feat: scanner

* feat: engine handles scan request

* feat: map projection index to column id

* feat: Impl record batch stream

* refactor: change BatchConverter to ProjectionMapper

* feat: add column_ids to mapper

* feat: implement SeqScan::build()

* chore: fix typo

* docs: add mermaid for ScanRegion

* style: fix clippy

* test: fix record batch test

* fix: update sequence and entry id

* test: test query

* feat: address CR comment

* chore: address CR comments

* chore: Update src/mito2/src/read/scan_region.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-08-28 06:59:31 +00:00
Lei, HUANG
6bf260a05c chore: write to mito2 (#2250) 
* chore: write to mito2

* fix: clippy

* feat: brdige memtable

* chore: rebase develop
 2023-08-25 06:18:42 +00:00
Lei, HUANG
dbe0e95f2f feat(mito2): concat and projection (#2243) 
* refactor: use arrow::compute::concat instead of push values to vector builders

* feat: support projection

* refactor: remove sequence

* refactor: concatenate

* fix: series must not be empty

* refactor: projection
 2023-08-25 03:25:27 +00:00
Bamboo1
48348aa364 fix: fix test_scheduler_continuous_stop in scheduler (#2252) 
* fix: fix test_scheduler_continuous_stop in scheduler

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add document annotation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

---------

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>
 2023-08-25 02:59:48 +00:00
Yingwen
4ee1034012 feat(mito): merge reader for mito2 (#2210) 
* feat: Implement slice and first/last timestamp for Batch

* feat(mito): implements sort/concat for Batch

* chore: fix typo

* chore: remove comments

* feat: sort and dedup

* test: test batch operations

* chore: cast enum to test op type

* test: test filter related api

* sytle: fix clippy

* feat: implement Node and CompareFirst

* feat: merge reader wip

* feat: merge wip

* feat: use batch's operation to sort and dedup

* feat: implement BatchReader for MergeReader

* feat: simplify codes

* test: test merge reader

* refactor: use test util to create batch

* refactor: remove unused imports

* feat: update comment

* chore: remove metadata() from Source

* chroe: update comment

* feat: source supports batch iterator

* chore: update comment
 2023-08-24 03:37:51 +00:00
Lei, HUANG
fdb5ad23bf refactor: use Batch::sort_and_dedup instead of Values::sort_in_place (#2235) 2023-08-23 08:56:49 +00:00
Bamboo1
4dbc32f532 refactor: remove associate type in scheduler to simplify it #2153 (#2194) 
* feature: add a simple scheduler using flume

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: only use a sender rather clone many senders

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: use select to avoid loop

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: add parameters in new function to build the flume capacity and number of receivers

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* test: add countdownlatch test concurrency

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* test: add barrier replacing countdownlatch to test concurrency and add wait all tasks finished in stop

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add some document annotation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add license header

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add Cargo.lock

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: Cargo.toml format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: delete println in test

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: add error handle

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: fix error handle and add test scheduler stop

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: spelling mistake

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: wait all tasks finished

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add todo which need wrap Future returned by send_async

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* test: remove unnessary sleep in test

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: resolve some conflicts

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: resolve conversation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: code format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: modify the function of schedule to synchronize and drop sender after stopping scheduler

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

---------

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>
 2023-08-23 06:28:00 +00:00
Yingwen
cc3e198975 feat(mito): Implement operations like concat and sort for Batch (#2203) 
* feat: Implement slice and first/last timestamp for Batch

* feat(mito): implements sort/concat for Batch

* chore: fix typo

* chore: remove comments

* feat: sort and dedup

* test: test batch operations

* chore: cast enum to test op type

* test: test filter related api

* sytle: fix clippy

* docs: comment for slice

* chore: address CR comment

Don't return Option in get_timestamp()/get_sequence()
 2023-08-22 12:03:02 +00:00
Yingwen
cd3755c615 feat(mito): Support handling RegionWriteRequest (#2218) 
* feat: convert region request to worker write request

* chore: remove unused codes

* test: fix tests compiler errors

* chore: remove create/close/open request from worker requests

* chore: add comment

* chore: fix typo
 2023-08-22 11:16:00 +00:00
Lei, HUANG
be1e13c713 feat(mito2): time series memtable (#2208) 
* feat: time series memtable

* feat: add some test

* fix: some clippy warnings

* chore: some rustdoc

* refactor: test

* fix: remove useless functions

* feat: add config for TimeSeriesMemtable

* chore: some optimize

* refactor: remove bucketing

* refactor: avoid cloing RegionMetadataRef across all Series; make initial_builder_capacity a const; sort batch only by timestamp and sequence
 2023-08-22 08:40:46 +00:00
Yingwen
4ba12155fe feat(mito): Implement SST format for mito2 (#2178) 
* chore: update comment

* feat: stream writer takes arrow's types

* feat: Define Batch struct

* feat: arrow_schema_to_store

* refactor: rename

* feat: write parquet in new format with tsids

* feat: reader support projection

* feat: Impl read compat

* refactor: rename SchemaCompat to CompatRecordBatch

* feat: changing sst format

* feat: make it compile

* feat: remove tsid and some structs

* feat: from_sst_record_batch wip

* chore: push array

* chore: wip

* feat: decode batches from RecordBatch

* feat: reader converts record batches

* feat: remove compat mod

* chore: remove some codes

* feat: sort fields by column id

* test: test to_sst_arrow_schema

* feat: do not sort fields

* test: more test helpers

* feat: simplify projection

* fix: projection indices is incorrect

* refactor: define write/read format

* test: test write format

* test: test projection

* test: test convert record batch

* feat: remove unused errors

* refactor: wrap get_field_batch_columns

* chore: clippy

* chore: fix clippy

* feat: build arrow schema from region meta in ReadFormat

* feat: initialize the parquet reader at `build()`

* chore: fix typo
 2023-08-17 06:25:50 +00:00
Ruihang Xia
4c3169431b feat: move region metadata to store-api (#2181) 
* add metadata & handle_read

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move metadata to store-api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* dep aquamarine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove temporary code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/store-api/Cargo.toml

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* remove old mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-08-16 04:18:26 +00:00
Yingwen
d4565c0a94 feat(mito): Defines the read Batch struct for mito2 (#2174) 
* feat: define batch

* feat: define Batch struct

* feat: stream writer takes arrow's types

* feat: make it compile

* feat: use uint64vector and uint8vector

* feat: add timestamps and primary key
 2023-08-15 06:39:21 +00:00
Ruihang Xia
2168970814 feat: define region server and related requests (#2160) 
* define region server and related requests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fill request body

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change mito2's request type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: bump greptime-proto to d9167cab (row insert/delete)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove name_to_index

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* address cr comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finilise

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-15 06:27:27 +00:00
Lei, HUANG
e924b44e83 refactor: KeyValues return ValueRef (#2170) 
* refactor: KeyValues return ValueRef

* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.

* feat: define min value of OpType as a const

* fix: toml format
 2023-08-14 14:51:13 +00:00
Lei, HUANG
55b5df9c51 feat: row wise converter (#2162) 
* feat: impl mem-comparable encoding for timestamp

* fix: test cases

* impl time series encode/decoder

* fix: merge unsupported match arms

* fix: clippy

* chore: big number delimiter

* feat: encode timestamps as i64

* fix: remove useless error variant
 2023-08-14 07:13:39 +00:00
Ruihang Xia
6d64e1c296 feat(mito): checkpoint for mito2 (#2142) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust dir structure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* downgrade log level

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply CR sugg.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add region id to log

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-13 09:26:01 +00:00
Yingwen
e6090a8d5b feat(mito): Write wal and memtable (#2135) 
* feat: hold wal entry in RegionWriteCtx

* feat: entry id and commited sequence

* feat: write to wal

* feat: write memtable

* feat: fill missing columns

* feat: validate write request

* feat: more validation to write request

* chore: fix typos

* feat: remove init and validate rows in new()

* style: fix clippy
 2023-08-12 07:44:44 +00:00
LFC
5c19913a91 build: on windows (#2054) 
* build on windows

* rebase develop

* fix: resolve PR comments
 2023-08-10 08:08:37 +00:00
Yingwen
d4e863882f feat: Add write method to memtable trait (#2123) 
* feat: validate semantic type

* feat: define KeyValues

* test: test semantic type check

* feat: impl KeyValues

* test: test KeyValues

* feat: Add write to Memtable

* style: fix clippy

* docs: more comment
 2023-08-09 04:07:50 +00:00
Ning Sun
c39de9072f refactor: use workspace dependencies for internal modules (#2119) 
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
 2023-08-08 11:02:34 +00:00
LFC
46fa3eb629 chore: upgrade rust toolchain to latest nightly (#2049) 
* chore: upgrade rust toolchain to latest nightly

* rebase develop

* update rust toolchain in ci
 2023-08-08 07:17:51 +00:00
Yingwen
cb4dd89754 feat(mito): Implement mito2 Wal (#2103) 
* feat: define wal struct

* feat: Implement Wal read/write

* feat: obsolete wal

* test: test wal

* refactor: use try_stream and remove async from scan
 2023-08-04 11:04:25 +00:00
shuiyisong
7efff2d704 fix: introduce taplo.toml and sort Cargo.toml (#2096) 
* fix: add taplo.toml

* fix: introduce taplo.toml & sort cargo.toml

* chore: remove option in ci too
 2023-08-04 06:44:45 +00:00
Yingwen
e5663a075f feat(mito): preparation to implementing write (#2085) 
* refactor: move request mod

* feat: add mutation

* feat: add handle_write mod

* feat: one mutation at a time

* feat: handle write requests

* feat: validate schema

* refactor: move schema check to write request

* feat: add convert value

* feat: fill default values

* chore: remove comments

* feat: remove code

* feat: remove code

* feat: buf requests

* style: fix clippy

* refactor: rename check functions

* chore: fix compile error

* chore: Revert "feat: remove code"

This reverts commit 6516597540.

* chore: Revert "feat: remove code"

This reverts commit 5f2b790a01.

* chore: upgrade greptime-proto

* chore: Update comment

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-08-04 02:53:02 +00:00
Yingwen
fdd4929c8f refactor(mito): mv mito2 request (#2086) 
* refactor: mv request mod to crate level

* refactor: mv SkippedFields
 2023-08-03 03:38:46 +00:00
Yingwen
c529c8a41b feat(mito): Implement open and close for mito2 regions (#2052) 
* feat: add close request

* feat: handle close and open request

* feat: Implement open

* test: add TestEnv::new

* feat: close region/engine and test

* style: fix clippy

* style: import log macros

* docs: update docs

* docs: add mermaid for manifest manager
 2023-08-01 10:49:07 +00:00
gongzhengyang
0eac56a442 chore: remove unused dependencies (#2055) 
Co-authored-by: gongzhengyang <gongzhengyang@bolean.com.cn>
 2023-08-01 07:43:03 +00:00
Yingwen
4d5ecb54c5 feat(mito): Implement open for RegionManifestManager (#2036) 
* feat: file purger trait

* feat: Implement open for RegionManifestManager

* feat: remove RegionVersion

* feat: Use RwLock

* chore: remove AtomicManifestVersion

* feat: Remove unused error

* feat: store meta action

* chore: update comment
 2023-07-31 10:04:22 +00:00
Ruihang Xia
f55bff51ac feat: set and retrieve trace id in log macro (#2016) 
* trace id passed by task local store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* modify log macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove tokio::spawn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use real trace id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-07-25 03:50:27 +00:00
Yingwen
5f65e3ff44 feat(mito): Port parquet writer and reader to mito2 (#2018) 
* feat(mito): Port Batch and BufferedWriter

* feat: encode metadata to parquet

* feat: define BatchReader trait

* chore: ParquetWriter write_all takes `&mut self`

* feat(mito): port ParquetReader

* chore: fix typo

* chore: address CR comment
 2023-07-24 09:35:21 +00:00
Yingwen
792d8dfe33 feat(mito): create region in mito2 engine (#1999) 
* chore: check table existence

* refactor: rename LevelMetaVec

* feat: create request to metadata

* refactor: Share MitoConfig between workers

* feat: impl handle_create_request

* refactor: move tests mod

* feat: validate time index nullable

* feat: test create region

* feat: test create if not exists

* feat: remove option

* style: fix clippy

* chore: address CR comments
 2023-07-21 06:41:34 +00:00
Yingwen
bb8468437e feat(mito): Define Version and metadata builders for mito2 (#1989) 
* feat: define structs for version

* feat: Build region from metadata and memtable builder

* feat: impl validate for metadata

* feat: add more fields to RegionMetadata

* test: more tests

* test: more check and test

* feat: allow overwriting version

* style: fix clippy
 2023-07-19 07:50:20 +00:00
Ruihang Xia
8bea853954 refactor(mito2): implement RegionManifestManager (#1984) 
* finilise manager and related API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl manifest initialize and update

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more test and utils

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-07-18 14:03:35 +00:00
Yingwen
ef7c5dd311 feat(mito): Implement WorkerGroup to handle requests (#1950) 
* feat: engine worker framework

* feat: worder comments

* feat: divide worker requests by type

* feat: handlers for worker thread

* refactor: rename requests to ddl and dml requests

* feat: methods to stop and submit requests

* refactor: rename request queue to request buffer

* refactor: remove ddl and dml request

* feat: send request to worker

* test: test stop

* docs(mito): worker group docs

* style: fix clippy

* docs: update WorkerGroup comment

* chore: address CR comments

* chore: fix comment issues

* feat: use mpsc::channel

* feat: check is_running flag

* chore: Add stop request to notify a worker

* refactor: add join_dir to join paths

* feat: redefine region requests

* docs: more comments

* refactor: rename worker thread to worker loop

* chore: address CR comments
 2023-07-14 08:06:44 +00:00