rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-22 16:00:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,707 Commits 131 Branches 253 Tags
6b39f5923decdd3be5b2af70ffcc06415585729f
Commit Graph

445 Commits

Author SHA1 Message Date
shuiyisong
007f7ba03c refactor: extract plugins crate (#2487) 
* chore: move frontend plugins fn

* chore: move datanode plugins to fn

* chore: add opt plugins

* chore: add plugins to meta-srv

* chore: setup meta plugins, wait for router extension

* chore: try use configurator for grpc too

* chore: minor fix fmt

* chore: minor fix fmt

* chore: add start meta_srv for hook

* chore: merge develop

* chore: minor fix

* chore: replace Arc<Plugins> with PluginsRef

* chore: fix header

* chore: remove empty file

* chore: modify comments

* chore: remove PluginsRef type alias

* chore: remove `OptPlugins`
 2023-10-09 04:54:27 +00:00
shuiyisong
067917845f fix: carry dbname from frontend to datanode (#2520) 
* chore: add dbname in region request header for tracking purpose

* chore: fix handle read

* chore: add write meter

* chore: add meter-core to dep

* chore: add converter between RegionRequestHeader and QueryContext & update proto version
 2023-10-08 06:30:23 +00:00
dennis zhuang
ff15bc41d6 feat: improve object storage cache (#2522) 
* feat: refactor object storage cache with moka

* chore: minor fixes

* fix: concurrent issues and invalidate cache after write/delete

* chore: minor changes

* fix: cargo lock

* refactor: rename

* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib

* fix: typo

* chore: style

* fix: toml format

* chore: toml

* fix: toml format

* Update src/object-store/src/layers/lru_cache/read_cache.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: update src/object-store/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: refactor and apply suggestions

* fix: typo

* feat: adds back allow list for caching

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* refactor: wrap inner Accessor with Arc

* chore: remove run_pending_task in read and write path

* chore: the arc is unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-10-08 03:27:49 +00:00
Weny Xu
f50f2a84a9 fix: open region missing options (#2473) 
* fix: open region missing options

* refactor: remove redundant clone

* chore: apply suggestions from CR

* chore: apply suggestions

* chore: apply suggestions

* test: add test for initialize_region_server

* feat: introduce RegionInfo
 2023-10-07 07:17:16 +00:00
Zhenchi
201acd152d fix: missing file engine with default options (#2519) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-28 10:25:12 +00:00
Zhenchi
7edafc3407 feat: push down filters to region engine (#2513) 
feat: pushdown filters to region engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-27 13:50:44 +00:00
Weny Xu
9282e59a3b fix: re-create heartbeat stream ASAP (#2499) 
* chore: set default connect_timeout_millis to 1000

* fix: re-create heartbeat stream ASAP

* chore: apply suggestions
 2023-09-27 04:00:16 +00:00
Ruihang Xia
515ce825bd feat: stack trace style debug print for error (#2489) 
* impl macro stack_trace_debug

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* manually mark external error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use debug print

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify the error and warn macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add docs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* replace snafu backtrace with location

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-26 11:23:21 +00:00
Zhenchi
0bf26642a4 feat: re-support query engine execute dml (#2484) 
* feat: re-support query engine execute dml

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: remove region_number in InsertRequest

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add doc comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-26 08:37:04 +00:00
Weny Xu
230a3026ad fix: dn doesn't have chance to send a heartbeat to the new leader (#2471) 
* refactor: set meta leader lease secs to 3s

* fix: correct default heartbeat interval

* refactor: ask meta leader in parallel

* feat: configure heartbeat client timeout to 500ms

* fix: trigger to send heartbeat immediately after fail

* fix: fix clippy
 2023-09-26 05:05:38 +00:00
LinFeng
621c6f371b feat: limit grpc message size (#2459) 
* feat: add two grpc config options

Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size

* test: add integration tests for message size limit
 2023-09-22 02:07:46 +00:00
Ruihang Xia
4c7ad44605 refactor: remove SqlStatementExecutor (#2464) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-22 01:57:48 +00:00
shuiyisong
20f4f7971a refactor: remove source and location in snafu display (#2428) 
* refactor: remove source pt 1

* refactor: remove source pt 2

* refactor: remove source pt 3

* refactor: remove location pt 1

* refactor: remove location pt 2

* chore: remove rustc files

* chore: fix error case

* chore: fix test

* chore: fix test

* chore: fix cr issue

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
 2023-09-21 02:55:24 +00:00
Ruihang Xia
34d6288945 feat: bring back sqlness and integration tests (#2448) 
* enable integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* disable sqlness region failover

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enable sqlness in CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort unstable result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* set require_lease_before_startup to true

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix inconsistent cache

* replace windows path chars

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore some integration cases in windows

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "ignore some integration cases in windows"

This reverts commit 122478b7c1.

* disable windows for now

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix close region bug in RegionHeartbeatResponseHandler

* disable failover tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-20 09:17:30 +00:00
Ruihang Xia
a5c499572c feat: open region in parallel (#2451) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-20 07:40:17 +00:00
Weny Xu
1fb2d95c5f fix: fix open region missing path (#2441) 
* fix: fix open region missing path

* fix: correct log

* chore: apply suggestions from CR

* fix: fix tests
 2023-09-19 08:50:59 +00:00
Weny Xu
43476e1ff9 refactor: rename coordination to require_lease_before_startup (#2431) 2023-09-18 11:07:42 +00:00
Weny Xu
c42cce57ca fix: fix incorrect matches (#2430) 
* fix: fix incorrect matches

* fix: fix incorrect status code
 2023-09-18 10:53:32 +00:00
Weny Xu
2c3ff90dbc feat: start services after first heartbeat response processed (#2424) 
* feat: start services after first heartbeat response processed

* refactor: watch changes in RegionAliveKeeper

* feat: add coordination to DatanodeOptions

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: enable coordination in sqlness
 2023-09-18 08:49:26 +00:00
Zhenchi
3a39215f11 feat: migrate file engine from table to reigon (#2365) 
* feat: migrate file engine from table to reigon

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/file-engine/src/engine.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: specify ts index for file engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: handle time index for external table

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: some integration testsg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add file schema and table schema compatibility

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: compatible file schema to region schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: simplify close

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: implement set_writable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: tests-integration compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local>
 2023-09-18 08:02:43 +00:00
Yingwen
4a82926d72 docs: fix cargo doc errors and warnings (#2421) 
* docs: fix cargo doc warnings and errors

* docs: fix warnings

* docs: fix warnings

* chore: rm src/common/function-macro/src/lib.rs
 2023-09-17 11:45:15 +00:00
Ruihang Xia
693e8de83a feat: scope spawned task with trace id (#2419) 
* feat: scope spawned task with trace id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-17 09:05:28 +00:00
JeremyHi
542e863ecc fix: missing datanode id on keep lease (#2415) 2023-09-17 07:57:17 +00:00
Weny Xu
5b08e03944 feat: sync regions between RegionServer and RegionAliveKeeper (#2417) 
* feat: sync regions between RegionServer and RegionAliveKeeper

* Apply suggestions from code review

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* refactor: rename event name

* chore: apply suggestions

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-09-17 07:55:44 +00:00
JeremyHi
98a40bae95 feat!: unify naming with options (#2416) 2023-09-17 07:24:57 +00:00
JeremyHi
342a6d071f feat: heartbeat request with header (#2412) 
* feat: heartbeat request with header

* chore: frontend send heartbeat with a longer interval
 2023-09-16 09:56:41 +00:00
dennis zhuang
627c5b7419 feat: move table operations from frontend to operator crate (#2411) 
* feat: move table operations from frontend to operator crate

* chore: blank line

* fix: toml format

* chore: move constants
 2023-09-16 07:58:45 +00:00
Ruihang Xia
5e35087b67 fix: generate region path with given prefix (#2409) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-16 03:23:36 +00:00
Ruihang Xia
c149c123c3 feat: reopen corresponding regions on starting datanode (#2399) 
* separate config and datanode impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* partial implement of fetching region id list

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reopen all regions on starting region server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness & assign default datanode id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* set writable on lease

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply cr suggs.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/datanode/src/datanode.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-15 13:30:20 +00:00
Vanish
0bd6b9bb39 feat: implement truncate region for mito2 (#2335) 
* feat: implement truncate region for mito2.

* chore: add license header and fix typos

* Update src/mito2/src/worker/handle_truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* cr

* chore: consider the flush task being executed before truncating the region.

* test

* feat: check flush and compaction tasks

* chore: remove useless changes

* Update src/mito2/src/manifest/action.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/mito2/src/worker/handle_flush.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: CR, consider sequence number

* test: use EventListener to test the flush task during truncate

* fix: fix listener error

* Update src/mito2/src/engine/truncate_test.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr

* fix: remove set None

* Update src/mito2/src/region/version.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/mito2/src/worker/handle_flush.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/mito2/src/worker/handle_truncate.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* doc: add some doc for FlushTruncateListener and RegionTruncate

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-09-15 13:20:01 +00:00
dennis zhuang
43e3c94fd1 refactor: catalog managers (#2405) 
* feat: rename catalog::local to catalog::memory

* refactor: catalog managers

* chore: license header
 2023-09-15 08:48:14 +00:00
LFC
fe5679e77e refactor: remove table ident (#2368) 
* refactor:
1. remove TableIdent, use TableId directly
2. use the latest greptime-proto
3. independently invalidate table id cache and table name cache

* rebase

* fix: resolve PR comments

* fix: resolve PR comments
 2023-09-15 05:14:40 +00:00
JeremyHi
d1adb915bf feat: set readonly first when deregister region (#2391) 
* feat: set readonly first when deregister region

* revert distxxx
 2023-09-14 12:12:38 +00:00
LFC
d08b05c963 fix: make test-integration able to compile (#2384) 
* fix: make test-integration able to compile

* chore: fmt toml

---------

Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-13 12:42:46 +00:00
JeremyHi
6f4779b474 feat: engine name in heartbeat (#2377) 2023-09-13 09:10:10 +00:00
Ruihang Xia
eeecce4623 refactor: remove table procedure (#2359) 
remove table procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
JeremyHi
80c5d52015 feat: stop region server (#2356) 
* feat: stop region server

* fix: close region first
 2023-09-12 07:57:15 -05:00
LFC
fe954b78a2 refactor: system tables in new region server (#2344) 
refactor: inverse the dependency between system tables and catalog manager
 2023-09-12 07:57:15 -05:00
JeremyHi
3cab6de391 feat: filter out empty heartbeat req (#2345) 
* feat: filter out empty heartbeat request

* fix: big mistake
 2023-09-12 07:57:15 -05:00
Weny Xu
f386329e29 refactor: introduce DdlTaskExecutor and refactor statement executor (#2341) 
* feat: add kv store option

* refactor: refactor statement executor

* refactor: refactor standalone table creator

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: move ShowCreateTable and CreateDatabase to StatementExecutor

* fix: fix RegionDistribution

* feat: build standalone

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2023-09-12 07:57:15 -05:00
Ruihang Xia
6215f124f7 refactor: remove datanode instance (#2342) 
* pass nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename region_alive_keepers

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
f287a5db9f feat: adapt region keep aliver for region server (#2333) 
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor, collapse one layer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove old heartbeat handler impls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove old region alive keeper

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove remote catalog manager

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test countdown task

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
eb7116ab56 feat: read/write works in distributed mode 🎉 (#2327) 
* add do_get method to RegionRequestHandler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move RegionRequestHandler to client crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use RegionRequestHandler in MergeScan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* minor fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
LFC
711e27d9fa feat: distributed alter table in region server (#2311) 
* feat: distributed alter table in region server

* rebase
 2023-09-12 07:57:15 -05:00
Zhenchi
7dde9ce3ce feat(frontend): migrate insert to region server (#2318) 
* feat(frontend): migrate insert to region server

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: move converter to Inserter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename convert function

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: add span id

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* retrigger action

* retrigger action

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-12 07:57:15 -05:00
Ruihang Xia
f71aa373c1 feat: start datanode with config (#2312) 
* remove memory-catalog and procedure

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* derive serde for MitoConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* start datanode with configs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dir in WalConfig

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add rename field attr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add stupid duplicated mito config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove wrong import

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wired compile error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-12 07:57:15 -05:00
dennis zhuang
b1599ad3a5 fix: can't adding new columns as primary key (#2310) 2023-09-12 07:57:15 -05:00
dennis zhuang
38697e0c4d feat: build http client for cloud object storage (#2314) 
* feat: build http client for s3/oss/azblob storages

* chore: style

* fix: test

* fix: cargo toml fmt
 2023-09-12 07:57:15 -05:00
Niwaka
3504d8254e fix: unused table options (#2267) 
* fix: unused table options keys

* refactor: simplify validate table options

* chore: Add newlines

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-09-12 07:57:15 -05:00
JeremyHi
fa542f6e93 feat: new heartbeat (#2299) 2023-09-12 07:57:15 -05:00