rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-05 21:02:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,566 Commits 105 Branches 225 Tags
60bdf9685f3f11496098f9f9ee71ea110e2e02b8
Commit Graph

1566 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruihang Xia
60bdf9685f feat: use the latest command line options for sqlness runner (#2371) 
feat: use the latest command line options

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-13 03:38:43 +00:00
Ruihang Xia
9c76d2cf54 feat: convert sql number to values with target type (#2370) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-13 11:14:42 +08:00
Weny Xu
1a7268186b chore: bump raft-engine to 22dfb4 (#2360) 2023-09-12 07:57:15 -05:00
Ruihang Xia
eeecce4623 refactor: remove table procedure (#2359) 
remove table procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
1ad5f6e5d5 refactor: system tables in FrontendCatalogManager (#2358) 
* rename method names

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove system table, table engine, register/deregister

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add system catalog

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* run nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* some documents

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix clippy

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-12 07:57:15 -05:00
Yingwen
46eca5026e fix(mito): Stores and recovers flushed sequence (#2355) 
* test: add test for reopen

* feat: last entry id starts from flushed entry id

* fix: store flushed sequence and recover it from manifest

* test: check sequence in alter test

* test: more tests for alter
 2023-09-12 07:57:15 -05:00
Weny Xu
912341e4fa fix: fix start issues under standalone mode (#2352) 
* fix: fix standalone starts

* chore: bump raft-engine to 571462e

* refactor: remove MetadataService
 2023-09-12 07:57:15 -05:00
JeremyHi
80c5d52015 feat: stop region server (#2356) 
* feat: stop region server

* fix: close region first
 2023-09-12 07:57:15 -05:00
Zhenchi
4af126eb1b feat: consolidate Insert request related partitioning and distributed processing operations into Inserter (#2346) 
* refactor: RegionRequest as param of RegionRequestHandler.handle

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: partition insert & delete reqs for both standalone and distributed mode

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: nit change

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: wrong function nameg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: do request in inserter & deleter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: remove RegionRequestHandler.handle

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: rename table_creator

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: nit change

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: nit change

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-12 07:57:15 -05:00
LFC
fe954b78a2 refactor: system tables in new region server (#2344) 
refactor: inverse the dependency between system tables and catalog manager
 2023-09-12 07:57:15 -05:00
JeremyHi
3cab6de391 feat: filter out empty heartbeat req (#2345) 
* feat: filter out empty heartbeat request

* fix: big mistake
 2023-09-12 07:57:15 -05:00
Yingwen
606ee43f1d feat(mito): Implement skeleton for alteration (#2343) 
* feat: impl handle_alter wip

* refactor: move send_result to worker.rs

* feat: skeleton for handle_alter_request

* feat: write requests should wait for alteration

* feat: define alter request

* chore: no warnings

* fix: remove memtables after flush

* chore: update comments and impl add_write_request_to_pending

* feat: add schema version to RegionMetadata

* feat: impl alter_schema/can_alter_directly

* chore: use send_result

* test: pull next_batch again

* feat: convert pb AlterRequest to RegionAlterRequest

* feat: validate alter request

* feat: validate request and alter metadata

* feat: allow none location

* test: test alter

* fix: recover files and flushed entry id from manifest

* test: test alter

* chore: change comments and variables

* chore: fix compiler errors

* feat: add is_empty() to MemtableVersion

* test: fix metadata alter test

* fix: Compaction picker doesn't notify waiters if it returns None

* chore: address CR comments

* test: add tests for alter request

* refactor: use send_result
 2023-09-12 07:57:15 -05:00
Lei, HUANG
3331e3158c feat(mito2): compaction (#2317) 
* feat: compaction component

* feat: mito2 compaction

* Avoid building time range predicates when merge SST files since in TWCS we don't enforce strict time window.

* fix: some CR comments

* minor: change CompactionRequest::senders to an option

* chore: handle compaction finish error

* feat: integrate compaction into region worker

* chore: rebase upstream

* fix: Some CR comments

* chore: Apply suggestions from code review

* style: fix clippy

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
Weny Xu
a4604afde5 refactor: rename NEXT_TABLE_ROUTE_PREFIX to TABLE_ROUTE_PREFIX (#2348) 
* refactor: rename NEXT_TABLE_ROUTE_PREFIX to TABLE_ROUTE_PREFIX

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Weny Xu
f386329e29 refactor: introduce DdlTaskExecutor and refactor statement executor (#2341) 
* feat: add kv store option

* refactor: refactor statement executor

* refactor: refactor standalone table creator

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor

* fix: fix RegionDistribution

* feat: build standalone

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Yingwen
3f6d557b8d feat: Implements a reader to make schema compatible (#2326) 
* docs: update comment

* feat: Add compat reader to SeqScan

* feat: add struct to compat pk and fields

* refactor: remove unused fields from ParquetReader

* feat: compat framework

* feat: Implement CompatPrimaryKey and CompatFields

* feat: implement compat reader

* feat: Test compat reader

* test: test compat reader

* feat: add more checks to concat

* style: fix clippy

* test: more tests for compat reader

* test: test reader with projection
 2023-09-12 07:57:15 -05:00
Ruihang Xia
6215f124f7 refactor: remove datanode instance (#2342) 
* pass nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename region_alive_keepers

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
LFC
1d83c942a9 refactor: script table creation (#2340) 
* refactor:
1. remove method `register_system_table` from CatalogManager
2. the creation of ScriptTable (as a system table) is removed from CatalogManager. Instead, the ScriptTable is created when Frontend instance is starting; and is created by calling Frontend instance's grpc handler.

* rebase
 2023-09-12 07:57:15 -05:00
Ruihang Xia
f287a5db9f feat: adapt region keep aliver for region server (#2333) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor, collapse one layer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove old heartbeat handler impls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove old region alive keeper

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove remote catalog manager

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test countdown task

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Zhenchi
dac6b2e80a feat(frontend): migrate delete to region server (#2329) 
* feat(frontend): migrate delete to region server

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add more check and do trim columns

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: RegionRequestHandler.handle retrun AffectedRows

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-12 07:57:15 -05:00
Yingwen
1e44e86d81 feat(mito): Stall write requests and add more flush tests (#2322) 
* feat: impl reject write

* feat: sanitize reject size

* feat: add should_stall to WriteBufferManager

* feat: stall requests

* test: mock WriteBufferManager

* feat: add new_with_manager for test and remove object_store from inner

* feat: add an event listener for tests

* feat: Use listener to test flush

* refactor: add flush_test.rs

* style: fix clippy

* feat: test write stall

* test: test flush empty
 2023-09-12 07:57:15 -05:00
JeremyHi
56691ff03b refactor: mailbox timeout (#2330) 
refactor: Optimize the timeout mechanism of the mailbox
 2023-09-12 07:57:15 -05:00
Weny Xu
e4de63625f refactor: refactor raft engine backend and state store (#2336) 
* refactor: remove redundant code

* refactor: refactor RaftEngineBackend Error to common_meta::error::Error

* refactor: refactor state store

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Ruihang Xia
4b2b59c31b refactor: clean unnecessary disabled lints (#2338) 
* clean manifest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean asscess_layer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean manifest manager

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean row_converter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean scheduler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean worker

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Weny Xu
2ee2d29085 refactor: move Sequence to common meta (#2337) 2023-09-12 07:57:15 -05:00
Yingwen
c3f6529178 fix: improve error message in validate_proto_value (#2328) 
* fix: correct error message in validate_proto_value()

* fix: print location in InvalidRequest error

* style: format
 2023-09-12 07:57:15 -05:00
Ruihang Xia
eb7116ab56 feat: read/write works in distributed mode 🎉 (#2327) 
* add do_get method to RegionRequestHandler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move RegionRequestHandler to client crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use RegionRequestHandler in MergeScan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* minor fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Zhenchi
5f7d48f107 feat(frontend): reorg insert converters and introduce stmt_to_region (#2324) 
* feat(frontend): reorg insert converters and introduce stmt_to_region

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: shorten import path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: add check for column count

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: clippy

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-12 07:57:15 -05:00
LFC
711e27d9fa feat: distributed alter table in region server (#2311) 
* feat: distributed alter table in region server

* rebase
 2023-09-12 07:57:15 -05:00
Weny Xu
922e342b63 refactor: refactor ddl manager (#2306) 
* refactor: refactor ddl manager

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Zhenchi
7dde9ce3ce feat(frontend): migrate insert to region server (#2318) 
* feat(frontend): migrate insert to region server

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: move converter to Inserter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename convert function

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: add span id

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* retrigger action

* retrigger action

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-12 07:57:15 -05:00
Yingwen
3eccb36047 feat: avoid using vector to get default value (#2323) 2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1 feat: start datanode with config (#2312) 
* remove memory-catalog and procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* derive serde for MitoConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* start datanode with configs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dir in WalConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add rename field attr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add stupid duplicated mito config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove wrong import

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wired compile error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
50fca2400e feat: adapt methods from RegionEngine for MitoEngine (#2315) 
* feat: adapt methods from RegionEngine for MitoEngine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* minor fixes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
JeremyHi
920763d7dd feat: add metric and manage tool for InactiveRegionKey (#2313) 
* feat: add metric and manage tool for InactiveRegionKey

* chore: by review comment
 2023-09-12 07:57:15 -05:00
dennis zhuang
a3d5931fca feat: unify all protocol options (#2316) 
* feat: unify all protocol options

* feat: adds enable to example configs

* chore: style

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
dennis zhuang
b1599ad3a5 fix: can't adding new columns as primary key (#2310) 2023-09-12 07:57:15 -05:00
dennis zhuang
38697e0c4d feat: build http client for cloud object storage (#2314) 
* feat: build http client for s3/oss/azblob storages

* chore: style

* fix: test

* fix: cargo toml fmt
 2023-09-12 07:57:15 -05:00
Yingwen
50220f8f04 feat: Impl write buffer manager for mito2 (#2309) 
* feat: add write buffer manager to builder

* feat: impl WriteBufferManager

* feat: impl MemtableVersion::mutable_usage

* chore: Address CR comments

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* refactor: rename mutable_limitation to mutable_limit

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
Niwaka
3504d8254e fix: unused table options (#2267) 
* fix: unused table options keys

* refactor: simplify validate table options

* chore: Add newlines

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
dennis zhuang
fad58835bf fix: don't raise an error when manifest directory is not created (#2308) 
* fix: don't raise an error when manifest directory is not created

* chore: apply suggestion

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-09-12 07:57:15 -05:00
Lei, HUANG
43fdff3639 feat: remove memtable request (#2307) 
* refactor: remove scan request from memtable API

* docs: Update comment

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
Weny Xu
271f80daad fix: LoadBase Selector cannot follow the region distribution rules (#2259) 
* fix: LoadBase Selector cannot follow the region distribution rules

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Lei, HUANG
36231a5d50 feat(mito2): add alloc_tracker for memtable (#2266) 
* feat: add alloc_tracker for memtable

* chore: integrate WriteBufferManager
 2023-09-12 07:57:15 -05:00
JeremyHi
a7fa40e16d fix: filter out outdated heartbeat (#2303) 
* fix: filter out outdated heartbeat, #1707

* feat: reorder handlers

* refactor: disableXXX to enableXXX

* feat: make full use of region leases to facilitate failover

* chore: minor refactor

* chore: by comment

* feat: logging on inactive/active
 2023-09-12 07:57:15 -05:00
Yingwen
648b2ae293 feat(mito): Flush region (#2291) 
* chore: call handle_flush_request

* feat: alias SchedulerRef and clean scheduler on drop

* feat: add scheduler to workers

* feat: remove RegionMemtableStats

* feat: pick regions to flush

* feat: add more fields to region flush task

* feat: smallvec workspace dep

* feat: Use list to hold immutable memtables

* feat: flush job wip

* feat: use access layer to read write sst

* feat: flush memtables to l0

* feat: write manifest

* feat: schedule next flush on success

* feat: schedule flush on success and failure

* feat: add purger to region

* feat: apply edit after flush

* feat: collect stats for SSTs

* feat: manual flush

* test: test flush and fix manifest test

* feat: remove flush scheduler job limit

* fix: typo

* style: clippy

* feat: clean flushed files on failure

* chore: address CR comment

* refactor: Use put_rows

* feat: Clean flush scheduler on drop

* feat: remove region flush status on drop and close

* chore: address CR comment
 2023-09-12 07:57:15 -05:00
Weny Xu
fa5e3b94d3 refactor: refactor ddl procedure (#2304) 2023-09-12 07:57:15 -05:00
Weny Xu
4818887e38 refactor: refactor DistInstance (#2305) 2023-09-12 07:57:15 -05:00
Ruihang Xia
eddff17523 feat: drop region in mito2 (#2286) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check in opening region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/mito2/src/worker/handle_drop.rs

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* remove file in order

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remove logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use scan to list files

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-12 07:57:15 -05:00
Weny Xu
c839ed271c refactor: refactor: ddl context (#2301) 
* refactor: refactor: ddl context

* refactor: remove unused code

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00