rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-16 13:00:40 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,495 Commits 133 Branches 252 Tags
8e9f2ffce42e2aa4fed1da166c0a649ed31c450d
Commit Graph

111 Commits

Author SHA1 Message Date
Weny Xu
7ff200c0fa fix: align region numbers to real regions (#2257) 2023-08-25 11:48:58 +00:00
Niwaka
a8f2e4468d feat: handle multiple grpc deletes (#2150) 
* feat: handle multiple grpc deletes

* fix: make DistDeleter::grpc_delete return usize

* fix: remove backtrace from MissingTimeIndexColumn

* fix: avoid using unwrap in PartitionRuleManager::split_delete_request

* fix: simplify MissingTimeIndexColumn
 2023-08-15 08:22:46 +00:00
Vanish
57836e762b feat: truncate table in standalone mode (#2090) 
* feat: impl table procedure in standalone mode

* chore: remove useless changes

* test: add some tests

* Update src/table-procedure/src/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* CR

* Update src/datanode/src/sql/truncate_table.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: fmt

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-08 11:23:36 +00:00
Ning Sun
c39de9072f refactor: use workspace dependencies for internal modules (#2119) 
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
 2023-08-08 11:02:34 +00:00
LFC
46fa3eb629 chore: upgrade rust toolchain to latest nightly (#2049) 
* chore: upgrade rust toolchain to latest nightly

* rebase develop

* update rust toolchain in ci
 2023-08-08 07:17:51 +00:00
Vanish
cf7e8c9142 feat: truncate region (#2097) 
* feat: impl truncate region

* test: test truncate region

* chore: typo

* refactor: table truncate

* chore: remove useless changes

* chore: reset version

* fix: wait for flush task to complete

* fix: clippy

* chore: remove useless changes

* CR

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/writer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* CR

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/manifest/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-04 12:26:25 +00:00
shuiyisong
7efff2d704 fix: introduce taplo.toml and sort Cargo.toml (#2096) 
* fix: add taplo.toml

* fix: introduce taplo.toml & sort cargo.toml

* chore: remove option in ci too
 2023-08-04 06:44:45 +00:00
Ruihang Xia
5bd80a74ab feat: prepare for implementing considering partition key in the distributed planner (#2000) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix frontend logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check substrait compatibility before pushdown

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* going to revert some rules

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test and clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove println

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-07-31 12:36:23 +00:00
Weny Xu
735c6390ca feat: implement alter table procedure (#1878) 
* feat: implement alter table procedure

* fix: fix uncaught error

* refactor: move fetch_table/s to table_routes.rs

* refactor: refactor error handling

* chore: apply suggestions from CR

* feat: switch to using alter table procedure

* feat: add table_version

* chore: apply suggestions from CR

* feat: introduce ddl_channel_manager

* chore: update greptime-proto
 2023-07-13 10:41:46 +08:00
Ben Baoyi
9ff7670adf refactor:remove common_error::prelude (#1946) 
* feat:Remove common_error::prelude

* fix merge error

* cr comment

* fix error
 2023-07-13 10:36:36 +08:00
Lei, HUANG
a7ea3bbc16 feat: manual compact api (#1912) 
* merge develop

* chore: merge develop

* fix: some cr commentx

* fix: cr comments
 2023-07-11 04:00:39 +00:00
Yingwen
f731193ddc refactor: Define RegionId as a new type (#1903) 
* refactor: Define RegionId as a new type

* chore: use into

* feat: custom debug print for region id

fix: test_show_create_table
 2023-07-07 21:26:03 +08:00
Weny Xu
ee16262b45 feat: add create table procedure (#1845) 
* feat: add create table procedure

* feat: change table_info type from vec u8 to RawTableInfo

* feat: return create table status

* fix: fix uncaught error

* refactor: use a notifier to respond to callers

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: add comment

* chore: apply suggestions from CR

* refacotr: make CreateMetadata step after DatanodeCreateTable step
 2023-07-04 22:24:43 +08:00
Lei, HUANG
3b6f70cde3 feat: initial twcs impl (#1851) 
* feat: initial twcs impl

* chore: rename SimplePicker to LeveledPicker

* rename some structs

* Remove Compaction strategy

* make compaction picker a trait object

* make compaction picker configurable for every region

* chore: add some test for ttl

* add some tests

* fix: some style issues in cr

* feat: enable twcs when creating tables

* feat: allow config time window when creating tables

* fix: some cr comments
 2023-07-04 16:42:27 +08:00
Vanish
563f6e05e2 feat: remove all the manifests in drop_region. (#1834) 
* feat: drop_region delete manifest file

* chore: remove redundant code

* chore: fmt

* chore: clippy

* chore: clippy

* feat: support delete_all in manifest.

* chore:CR

* test: test_drop_basic, test_drop_reopen

* chore: cr

* fix: typo

* chore: cr
 2023-06-30 17:42:11 +08:00
Ruihang Xia
c77b94650c refactor: remove Table::scan method (#1855) 
* remove scan method

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-30 12:13:14 +08:00
Lei, HUANG
f287d3115b chore: replace result assertions (#1840) 
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g

* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
 2023-06-27 19:14:48 +08:00
LFC
fcff66e039 chore: deny unused results (#1825) 
* chore: deny unused results

* rebase
 2023-06-27 15:33:53 +08:00
Yingwen
fd412b7b07 refactor!: Uses table id to locate tables in table engines (#1817) 
* refactor: add table_id to get_table()/table_exists()

* refactor: Add table_id to alter table request

* refactor: Add table id to DropTableRequest

* refactor: add table id to DropTableRequest

* refactor: Use table id as key for the tables map

* refactor: use table id as file engine's map key

* refactor: Remove table reference from engine's get_table/table_exists

* style: remove unused imports

* feat!: Add table id to TableRegionalValue

* style: fix cilppy

* chore: add comments and logs
 2023-06-25 15:05:20 +08:00
Lei, HUANG
7961de25ad feat: persist compaction time window (#1757) 
* feat: persist compaction time window

* refactor: remove useless compaction window fields

* chore: revert some useless change

* fix: some CR comments

* fix: comment out unstable sqlness test

* revert commented sqlness
 2023-06-13 10:15:42 +08:00
Ruihang Xia
8c78368374 refactor: replace #[snafu(backtrace)] with Location (#1753) 
* remove snafu backtrace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-12 11:55:33 +08:00
dennis zhuang
f08f726bec test: s3 manifest (#1755) 
* feat: change default manifest options

* test: s3 manifest

* feat: revert checkpoint_margin to 10

* Update src/object-store/src/test_util.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-06-09 10:28:41 +00:00
Yingwen
1b4976b077 feat: Adds some metrics for write path and flush (#1726) 
* feat: more metrics

* feat: Add preprocess elapsed

* chore(storage): rename metric

* test: fix tests
 2023-06-05 21:35:44 +08:00
Ruihang Xia
e5b6f8654a feat: optimizer rule to pass expected output ordering hint (#1675) 
* move type convertsion rule into optimizer dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement order_hint rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* it works!

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use column name instead

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* accomplish test case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update lock file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-02 03:43:51 +00:00
Zheming Li
5467ea496f feat: Add column supports at first or after the existing columns (#1621) 
* feat: Add column supports at first or after the existing columns

* Update src/common/query/Cargo.toml

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-06-01 02:13:00 +00:00
dennis zhuang
ae8203fafa fix: prepare statement doesn't support insert clause (#1680) 
* fix: insert clause doesn't support prepare statement

* fix: manifeste dir

* fix: format

* fix: temp path
 2023-05-31 20:14:58 +08:00
Weny Xu
0460f3ae30 test: add write test for region failover (#1673) 
* test: add write test for region failover

* test: add written data assertion after failover

* test: support more storage types
 2023-05-31 15:42:00 +08:00
Lei, HUANG
72b6bd11f7 feat: adapt window reader to order rules (#1671) 
* feat: adapt window reader to order rules

* fix: add asc sort test case
 2023-05-31 03:36:17 +00:00
Ruihang Xia
b27c569ae0 refactor: add scan_to_stream() to Table trait to postpone the stream generation (#1639) 
* add scan_to_stream to Table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl parquet stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reorganise adapters

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement scan_to_stream for mito table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add location info

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: table scan

* UT pass

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl project record batch

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix information schema

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove one todo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix errors generated by merge commit

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add output_ordering method to record batch stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rustfmt

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enhance error types

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
 2023-05-29 20:03:47 +08:00
Weny Xu
732784d3f8 feat: support to load missing region (#1651) 
* feat: support to load missing region

* Update src/mito/src/table.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-05-26 03:30:46 +00:00
Weny Xu
3dc45f1c13 feat: implement CloseRegionHandler (#1569) 
* feat: implement CloseRegionHandler

* feat: register heartbeat response handlers

* test: add tests for heartbeat response handlers

* fix: drop table does not release regions

* chore: apply suggestion from CR

* fix: fix close region issue

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: modify method name and add log

* refactor: refactor HeartbeatResponseHandler

* chore: apply suggestion from CR

* refactor: remove close method from Region trait

* chore: apply suggestion from CR

* chore: remove PartialEq from CloseTableResult

* chore: apply suggestion from CR
 2023-05-23 15:44:27 +08:00
dennis zhuang
7c55783e53 feat!: reorganize the storage layout (#1609) 
* feat: adds data_home to DataOptions

* refactor: split out object store stuffs from datanode instance

* feat: move data_home into FileConfig

* refactor: object storage layers

* feat: adds datanode path to procedure paths

* feat: temp commit

* refactor: clean code

* fix: forgot files

* fix: forgot files

* Update src/common/test-util/src/ports.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update tests/runner/src/env.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: compile error

* chore: cr comments

* fix: dependencies order in cargo

* fix: data path in test

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-05-23 13:58:26 +08:00
Weny Xu
9f67ad8bce fix: fix doesn't release closed regions issue (#1596) 
* fix: fix close region issue

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* refactor: remove close method from Region trait

* chore: remove PartialEq from CloseTableResult
 2023-05-23 11:40:12 +08:00
Lei, HUANG
8e7ec4626b refactor: remove useless error (#1624) 
* refactor: remove useless

* fix: remove useless error variant
 2023-05-22 22:55:27 +08:00
Yingwen
32ad358323 fix(table-procedure): Open table in RegisterCatalog state (#1617) 
* fix(table-procedure): on_register_catalog should use open_table

* test: Test recover RegisterCatalog state

* test: Fix subprocedure does not execute in test

* feat(mito): adjust procedure log level

* refactor: rename execute_parent_procedure

execute_parent_procedure -> execute_until_suspended_or_done
 2023-05-22 17:54:02 +08:00
Vanish
8764ce7845 feat: add delete WAL in drop_region (#1577) 
* feat: add delete WAL in drop_region

* chore: fix typo err.

* feat: mark all SSTs deleted and remove the region from StorageEngine's region map.

* test: add test_drop_region for StorageEngine.

* chore: make clippy happy

* fix: fix conflict

* chore: CR.

* chore: CR

* chore: fix clippy

* fix: temp file life time
 2023-05-18 18:02:34 +08:00
Weny Xu
ca75a7b744 fix: remove region number validation (#1593) 
* fix: remove region number validation

* Update src/mito/src/engine.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-05-17 09:23:56 +08:00
WU Jingdi
fb1ac0cb9c feat: support user config manifest compression (#1579) 
* feat: support user config manifest compression

* chore: change style

* chore: enhance test
 2023-05-16 11:02:59 +08:00
Yingwen
8d54d40b21 feat: Add FlushPicker to flush regions periodically (#1559) 
* feat: Add FlushPicker

* feat(storage): Add close to StorageEngine

* style(storage): fix clippy

* feat(storage): Close regions in StorageEngine::close

* chore(storage): Clear requests on scheduler stop

* test(storage): Test flush picker

* feat(storage): Add metrics for auto flush

* feat(storage): Add flush reason and record it in metrics

* feat: Expose flush config

docs(config): Update config example

* refactor(storage): Run auto flush task in FlushScheduler

* refactor(storage): Add FlushItem trait to make FlushPicker easy to test
 2023-05-15 17:29:28 +08:00
Weny Xu
66903d42e1 feat: implement OpenTableHandler (#1567) 
* feat: implement OpenTableHandler

* chore: apply suggestion from CR

* chore: apply suggestion from CR
 2023-05-15 10:47:28 +08:00
discord9
2c82ded975 feat: table metrics (#1469) 
* feat: Statistic

* add todo

* fmt: cargo fmt

* feat: some simple impl for MemTable

* chore: a try on adding statistics

* Update src/table/src/stats.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* docs: fix typo

* newlines unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-05-06 14:59:49 +08:00
Yingwen
51be35a7b1 feat(mito): Combine the original and procedure's implementation (#1468) 
* fix(mito): Add metrics to mito DDL procedure

* feat(mito): Use procedure's implementation to create table

* feat(mito): Use procedure's implementation to alter table

* feat(mito): Use procedure's implementation to drop table

* style(mito): Fix clippy

* test(mito): Fix tests

* feat(mito): Add TableCreator

* feat(mito): update alter table procedure

* fix(mito): alter procedure create alter op first

* feat(mito): Combine alter table code

* fix(mito): Fix deadlock

* feat(mito): Simplify drop table procedure
 2023-04-28 11:48:52 +08:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448) 
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
 2023-04-25 12:04:02 +08:00
Zheming Li
c75845c570 fix: wrong next column in manifest (#1440) 
Signed-off-by: Zheming Li <nkdudu@126.com>
 2023-04-23 11:25:38 +08:00
Yingwen
9cd2cf630d feat: procedures for file table engine (#1417) 
* refactor: Add table_ref() to requests as their methods

* feat: Add CreateImmutableFileTable

* feat: Add DropImmutableFileTable

* feat: Implement TableEngineProcedure for ImmutableFileTableEngine

* feat: Add common-procedure-test crate

* refactor: mito engine use common-procedure-test to test procedures

* test: Add test for create and drop table

* chore: Address review comments
 2023-04-20 18:52:44 +08:00
shuiyisong
145f8eb5a7 refactor: parallelize open table (#1392) 
* refactor: change open_table to parallel on datanode startup

* chore: try move out register schema table

* chore: change mito engine to key lock

* chore: minor change

* chore: minor change

* chore: update error definition

* chore: remove rwlock on tables

* chore: try parallel register table on schema provider

* chore: add rt log

* chore: add region open rt log

* chore: add actual open region rt log

* chore: add recover rt log

* chore: divide to three part rt log

* chore: remove debug log

* chore: add replay rt log

* chore: update cargo lock

* chore: remove debug log

* chore: revert unused change

* chore: update err msg

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fix cr issue

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fix cr issue

* chore: fix cr issue

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-04-18 21:36:29 +08:00
Lei, HUANG
68e64a6ce9 feat: add some metrics (#1384) 
* feat: add some metrics

* fix: compile errors
 2023-04-14 20:46:45 +08:00
Yingwen
d988b43996 feat: Add drop table procedure to mito (#1377) 
* feat: Add drop table procedure to mito

* feat: remove table from engine and then close it
 2023-04-14 13:09:38 +08:00
Ruihang Xia
6b6617f9cb build: specify clippy denies in cargo config (#1351) 
* build: specify clippy denies in cargo config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* deny implicit clone

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-04-11 09:48:52 +00:00
Lei, HUANG
3e8ec8b73a fix: avoid panic when no region found in table (#1359) 2023-04-11 16:58:18 +08:00