rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-25 01:10:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,840 Commits 131 Branches 253 Tags
a691cff0c4286570ed0095175faa1cb51eefbeb1
Commit Graph

148 Commits

Author SHA1 Message Date
Ruihang Xia
a691cff0c4 feat: alter logical region in metric region (#2726) 
* add test for add and alter logical region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove table id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* CR sugg.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* extract internal states into a struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused deps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* tweak fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-11-15 03:52:52 +00:00
Weny Xu
dc46e96879 refactor: replace InactiveRegionManager with RegionLeaseKeeper (#2729) 
* refactor: replace InactiveRegionManager with RegionLeaseKeeper

* refactor: remove InactiveRegionManager

* chore: remove remark_inactive_region field

* chore: apply suggestions from CR

* refactor: reduce meta server cache scope

* chore: remove redundant code
 2023-11-14 05:16:13 +00:00
Weny Xu
a3611516a2 refactor: heartbeat response contains region role (#2718) 2023-11-09 09:24:36 +00:00
Ruihang Xia
93f21b188d chore: unify workspace dependencies and remove some unused (#2713) 
* log-store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* base64

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* udeps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change the workspace deps' style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-09 07:34:22 +00:00
Ruihang Xia
6fd04e38a3 feat: implement create region request for metric engine (#2694) 
* implement basic put/get/exist interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test add_column and add_table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move engine test to test_util

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* verify incoming create region request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* create data region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply review sugg

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-06 03:48:26 +00:00
Ruihang Xia
0dca63bc7b feat: Metric engine skeleton (#2687) 
* metadata region logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix mito2 test feature gate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* override unused warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add basic test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply review sugg

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-03 08:34:08 +00:00
Weny Xu
ce867fb583 feat: introduce the region lease keeper (#2645) 
* feat: introduce the region lease keeper

* chore: apply suggestions from CR

* refactor: simplify `retain_active_regions`

* refactor: remove Default of RegionStat

* chore: add todo comments

* chore: apply suggestions from CR

* refactor: simplify `retain_active_regions`

* fix: fix ci
 2023-11-02 12:01:17 +00:00
Weny Xu
54ed7529ca feat: introduce the region role (#2640) 
* feat: introduce region role

* chore: apply suggestions from CR
 2023-10-31 06:59:22 +00:00
Wei
36c0742c45 feat: region disk usage statistic (#2665) 
* feat: region usage

* chore: code comment

* chore: unit test

* feat: region usage for engine

* chore: rename to region usage

* feat: add to heartbeat

* chore: u64 to i64 convert

* chore: cr comment.

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/file-engine/src/engine.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-31 03:57:37 +00:00
Ruihang Xia
5d8b0e8154 chore: bump dependencies (#2659) 
* fix all hard error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* trivial changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix order by

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sql keyword and data type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix range exec's input partitioning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix cover input type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix explain analyze

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update distributed mode sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update locks

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* downgrade dlv-list

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-26 12:24:14 +00:00
JeremyHi
ae95f23e05 feat: add metrics for region server (#2552) 
* feat: add metircs for region server

* fix: add comment and remove unused code
 2023-10-10 07:40:16 +00:00
Yingwen
8a5ef826b9 fix(mito): Do not write to memtables if writing wal is failed (#2561) 
* feat: add writes total metrics

* fix: don't write memtable if write ctx is failed

* feat: write rows metrics
 2023-10-10 06:55:57 +00:00
Yingwen
d4577e7372 feat(mito): add metrics to mito engine (#2556) 
* feat: allow discarding a timer

* feat: flush metrics

* feat: flush bytes and region count metrics

* refactor: add as_str to get static string

* feat: add handle request elapsed metrics

* feat: add some write related metrics

* style: fix clippy
 2023-10-10 03:53:17 +00:00
Yingwen
0593c3bde3 fix(mito): pruning for mito2 (#2525) 
* fix: pruning for mito2

* chore: refactor projection parameters; add some tests; customize row group size for each flush task.

* chore: pass whole RegionFlushRequest

---------

Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
 2023-10-08 03:45:15 +00:00
Bamboo1
b44e39f897 feat: the schema of RegionMetadata is not output during debug (#2498) 
* feat: the schema of RegionMetadata is not output during debug because column_metadatas contains duplicate information

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: the id_to_index of RegionMetadata is not output during debug

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: add debug trait

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* feat: use default debug in ConcreteDataType

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: add std::fmt

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* test: add debug trait test

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: typo

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: resolve conversation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: format

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: test bug

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

---------

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>
 2023-10-07 08:01:54 +00:00
Ruihang Xia
515ce825bd feat: stack trace style debug print for error (#2489) 
* impl macro stack_trace_debug

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* manually mark external error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use debug print

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify the error and warn macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add docs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* replace snafu backtrace with location

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-26 11:23:21 +00:00
Vanish
7fc9604735 feat: distribute truncate table in region server (#2414) 
* feat: distribute truncate table

* chore: add metrics for truncate table

* test: add sqlness test

* chore: cr

* test: add multi truncate

* chore: add trace id to the header
 2023-09-26 11:14:14 +00:00
Wei
aef9e7bfc3 refactor: not allowed int64 type as time index (#2460) 
* refactor: remove is_timestamp_compatible.

* chore: fmt

* refactor: remove int64 to timestamp match

* chore

* chore: apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fmt

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-09-22 06:28:02 +00:00
Yingwen
c9f8b9c7c3 feat: update proto and remove create_if_not_exists (#2467) 2023-09-22 03:24:49 +00:00
Ruihang Xia
e3f53a8060 fix: add slash after generated region_dir (#2463) 
* fix: add slash after generated region_dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-21 07:46:05 +00:00
shuiyisong
20f4f7971a refactor: remove source and location in snafu display (#2428) 
* refactor: remove source pt 1

* refactor: remove source pt 2

* refactor: remove source pt 3

* refactor: remove location pt 1

* refactor: remove location pt 2

* chore: remove rustc files

* chore: fix error case

* chore: fix test

* chore: fix test

* chore: fix cr issue

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
 2023-09-21 02:55:24 +00:00
Yingwen
17e560c909 feat(mito): Allow to retry create request and alter request (#2447) 
* feat: RegionMetadataBuilder allow adding/dropping columns multiple times

* test: test add if not exists/drop if exists

* feat: change validator and add need_alter

* test: fix tests and test need_alter

* test: test alter retry

* feat: open before create

* style: fix clippy
 2023-09-20 06:36:46 +00:00
Ruihang Xia
fbbf3978d9 fix: render comment in SHOW CREATE TABLE (#2427) 
* feat: add comment field to ColumnDef

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-18 10:51:10 +00:00
Zhenchi
3a39215f11 feat: migrate file engine from table to reigon (#2365) 
* feat: migrate file engine from table to reigon

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/file-engine/src/engine.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: specify ts index for file engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: handle time index for external table

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: some integration testsg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add file schema and table schema compatibility

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: compatible file schema to region schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: simplify close

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: implement set_writable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: tests-integration compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local>
 2023-09-18 08:02:43 +00:00
Yingwen
4a82926d72 docs: fix cargo doc errors and warnings (#2421) 
* docs: fix cargo doc warnings and errors

* docs: fix warnings

* docs: fix warnings

* chore: rm src/common/function-macro/src/lib.rs
 2023-09-17 11:45:15 +00:00
dennis zhuang
627c5b7419 feat: move table operations from frontend to operator crate (#2411) 
* feat: move table operations from frontend to operator crate

* chore: blank line

* fix: toml format

* chore: move constants
 2023-09-16 07:58:45 +00:00
Vanish
0bd6b9bb39 feat: implement truncate region for mito2 (#2335) 
* feat: implement truncate region for mito2.

* chore: add license header and fix typos

* Update src/mito2/src/worker/handle_truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* cr

* chore: consider the flush task being executed before truncating the region.

* test

* feat: check flush and compaction tasks

* chore: remove useless changes

* Update src/mito2/src/manifest/action.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/mito2/src/worker/handle_flush.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: CR, consider sequence number

* test: use EventListener to test the flush task during truncate

* fix: fix listener error

* Update src/mito2/src/engine/truncate_test.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr

* fix: remove set None

* Update src/mito2/src/region/version.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/mito2/src/worker/handle_flush.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/mito2/src/worker/handle_truncate.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* doc: add some doc for FlushTruncateListener and RegionTruncate

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-09-15 13:20:01 +00:00
Bamboo1
6aec30a1a8 feat: reserve internal column (#2396) 
* feat: reserve internal column

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* test: add function test

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* fix: spell typos

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

* chore: resolve conversation

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>

---------

Signed-off-by: ZhuZiyi <zyzhu2001@gmail.com>
 2023-09-15 11:19:40 +00:00
JeremyHi
9572b1edbb feat: region storage path (#2404) 
* feat: region storage path

* Update src/common/meta/src/key/datanode_table.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: by cr

* feat: upgrade proto

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2023-09-15 09:07:54 +00:00
Yingwen
a7df5a7c9a fix(mito): incorrect field index in ProjectionMapper (#2388) 
* chore: update todo comments

* test: add test for projection

* fix: panics when projecting fields

* chore: remove todos
 2023-09-14 04:15:15 +00:00
Yingwen
d4ae8a6fed feat(mito): Add writable flag to region (#2349) 
* feat: add writable flag to region.

* refactor: rename MitoEngine to MitoEngine::scanner

* feat: add set_writable() to RegionEngine

* feat: check whether region is writable

* feat: make set_writable sync

* test: test set_writable

* docs: update comments

* feat: send result on compaction failure

* refactor: wrap output sender in new type

* feat: on failure

* refactor: use get_region_or/writable_region_or

* refactor: remove send_result

* feat: notify waiters on flush scheduler drop

* test: fix tests

* fix: only alter writable region
 2023-09-14 02:45:30 +00:00
Yingwen
da54a0c139 fix: alter table procedure forgets to update next column id (#2385) 
* feat: add more info to error messages

* feat: store next column id in procedure

* fix: update next column id for table info

* test: fix add col test

* chore: remove location from invalid request error

* test: update test

* test: fix test
 2023-09-14 02:06:57 +00:00
JeremyHi
80c5d52015 feat: stop region server (#2356) 
* feat: stop region server

* fix: close region first
 2023-09-12 07:57:15 -05:00
Yingwen
606ee43f1d feat(mito): Implement skeleton for alteration (#2343) 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
 2023-09-12 07:57:15 -05:00
Lei, HUANG
3331e3158c feat(mito2): compaction (#2317) 
* feat: compaction component

* feat: mito2 compaction

* Avoid building time range predicates when merge SST files since in TWCS we don't enforce strict time window.

* fix: some CR comments

* minor: change CompactionRequest::senders to an option

* chore: handle compaction finish error

* feat: integrate compaction into region worker

* chore: rebase upstream

* fix: Some CR comments

* chore: Apply suggestions from code review

* style: fix clippy

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
LFC
711e27d9fa feat: distributed alter table in region server (#2311) 
* feat: distributed alter table in region server

* rebase
 2023-09-12 07:57:15 -05:00
Ruihang Xia
46d171d341 chore: bump greptime-proto to replace region_dir (#2290) 
chore: bump greptime-proto to replace region_dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
6b8cf0bbf0 feat: impl region engine for mito (#2269) 
* update proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* convert request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update proto

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* import result convertor

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename symbols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-28 09:24:12 +00:00
Yingwen
e2522dff21 feat(mito): Skeleton for scanning a region (#2230) 
* feat: define stream builder

* feat: scan region wip

* feat: create SeqScan in ScanRegion

* feat: scanner

* feat: engine handles scan request

* feat: map projection index to column id

* feat: Impl record batch stream

* refactor: change BatchConverter to ProjectionMapper

* feat: add column_ids to mapper

* feat: implement SeqScan::build()

* chore: fix typo

* docs: add mermaid for ScanRegion

* style: fix clippy

* test: fix record batch test

* fix: update sequence and entry id

* test: test query

* feat: address CR comment

* chore: address CR comments

* chore: Update src/mito2/src/read/scan_region.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-08-28 06:59:31 +00:00
Yingwen
cd3755c615 feat(mito): Support handling RegionWriteRequest (#2218) 
* feat: convert region request to worker write request

* chore: remove unused codes

* test: fix tests compiler errors

* chore: remove create/close/open request from worker requests

* chore: add comment

* chore: fix typo
 2023-08-22 11:16:00 +00:00
Lei, HUANG
be1e13c713 feat(mito2): time series memtable (#2208) 
* feat: time series memtable

* feat: add some test

* fix: some clippy warnings

* chore: some rustdoc

* refactor: test

* fix: remove useless functions

* feat: add config for TimeSeriesMemtable

* chore: some optimize

* refactor: remove bucketing

* refactor: avoid cloing RegionMetadataRef across all Series; make initial_builder_capacity a const; sort batch only by timestamp and sequence
 2023-08-22 08:40:46 +00:00
Zhenchi
87a730658a refactor: add ThinTable to proxy tables from infoschema (#2193) 
* refactor: add thin table to proxy tables in info_schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix(catalog): fix typo in DataSourceAdapter struct name

* fix: remove redundant Send + Sync

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor(catalog): rename DataSourceAdapter to InformationTableDataSource

* feat(catalog): add ThinTableAdapter for adapting ThinTable to Table interface

* rebase develop

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: default impl for table_type of InformationTable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: filter_pushdown as table field

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove explicit type declaration

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-08-17 15:19:14 +00:00
Ruihang Xia
4aaf6aa51b feat: implement query API for RegionServer (#2197) 
* some initial change

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl dummy structs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* decode and send logical plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement table scan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add some comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-17 11:02:31 +00:00
Yingwen
4ba12155fe feat(mito): Implement SST format for mito2 (#2178) 
* chore: update comment

* feat: stream writer takes arrow's types

* feat: Define Batch struct

* feat: arrow_schema_to_store

* refactor: rename

* feat: write parquet in new format with tsids

* feat: reader support projection

* feat: Impl read compat

* refactor: rename SchemaCompat to CompatRecordBatch

* feat: changing sst format

* feat: make it compile

* feat: remove tsid and some structs

* feat: from_sst_record_batch wip

* chore: push array

* chore: wip

* feat: decode batches from RecordBatch

* feat: reader converts record batches

* feat: remove compat mod

* chore: remove some codes

* feat: sort fields by column id

* test: test to_sst_arrow_schema

* feat: do not sort fields

* test: more test helpers

* feat: simplify projection

* fix: projection indices is incorrect

* refactor: define write/read format

* test: test write format

* test: test projection

* test: test convert record batch

* feat: remove unused errors

* refactor: wrap get_field_batch_columns

* chore: clippy

* chore: fix clippy

* feat: build arrow schema from region meta in ReadFormat

* feat: initialize the parquet reader at `build()`

* chore: fix typo
 2023-08-17 06:25:50 +00:00
Ruihang Xia
4c3169431b feat: move region metadata to store-api (#2181) 
* add metadata & handle_read

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move metadata to store-api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* dep aquamarine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove temporary code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/store-api/Cargo.toml

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* remove old mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-08-16 04:18:26 +00:00
Ruihang Xia
2168970814 feat: define region server and related requests (#2160) 
* define region server and related requests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fill request body

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change mito2's request type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: bump greptime-proto to d9167cab (row insert/delete)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove name_to_index

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* address cr comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finilise

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-08-15 06:27:27 +00:00
Lei, HUANG
e924b44e83 refactor: KeyValues return ValueRef (#2170) 
* refactor: KeyValues return ValueRef

* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.

* feat: define min value of OpType as a const

* fix: toml format
 2023-08-14 14:51:13 +00:00
Ning Sun
c39de9072f refactor: use workspace dependencies for internal modules (#2119) 
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
 2023-08-08 11:02:34 +00:00
LFC
46fa3eb629 chore: upgrade rust toolchain to latest nightly (#2049) 
* chore: upgrade rust toolchain to latest nightly

* rebase develop

* update rust toolchain in ci
 2023-08-08 07:17:51 +00:00
Vanish
cf7e8c9142 feat: truncate region (#2097) 
* feat: impl truncate region

* test: test truncate region

* chore: typo

* refactor: table truncate

* chore: remove useless changes

* chore: reset version

* fix: wait for flush task to complete

* fix: clippy

* chore: remove useless changes

* CR

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/writer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* CR

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/manifest/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-04 12:26:25 +00:00