rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-08 22:32:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,020 Commits 103 Branches 226 Tags
8ce8a8f3c7681872c997cb5f6c104f6805da9ec7
Commit Graph

2020 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ruihang Xia
8ce8a8f3c7 fix: revert unfinished route table change (#3008) 
* Revert "refactor: hide `RegionRoute` behind `TableRouteValue` (#2989)"

This reverts commit 1641fd572a.

* Revert "feat: MetricsEngine table route (part 1) (#2952)"

This reverts commit 6ac47e939c.
 2023-12-26 09:56:49 +00:00
Ruihang Xia
bf635a6c7c feat: replace ahash with murmur3 on generating tsid (#3007) 
* feat: replace ahash with murmur3 on generating tsid

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change random seed

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-26 07:45:31 +00:00
LFC
196c06db14 feat: make logging to stdout configurable (#3003) 
* feat: make logging to stdout configurable

* fix sqlness

* fix: resolve PR comments
 2023-12-26 07:37:50 +00:00
Ruihang Xia
99565a3676 fix: update doc label on pr edit (#3005) 
* fix: update doc label on pr edit

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update token

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more trigger types

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-26 14:53:34 +08:00
Weny Xu
c902d43380 refactor(remote_wal): add StandaloneWalConfig (#3002) 
* feat: integrate remote wal to standalone

* fix: test

* refactor: refactor standalone wal config

* chore: change default kafka port to 9092

* chore: apply suggestions from CR

---------

Co-authored-by: niebayes <niebayes@gmail.com>
 2023-12-26 04:31:41 +00:00
Wei
95f172eb81 feat: convert FileMetaData to ParquetMetaData (#2980) 
* feat: can convert Format FileMetaData to ParquetMetaData

* test: parquet metadata equal

* chore: test

* chore: cr comment

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

* chore: cr comment

* refactor: type name

* chore: cr comment

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-12-26 04:27:36 +00:00
Ruihang Xia
3bd2f79841 chore(ci): auto doc labeler job (#2998) 
* chore(ci): auto doc labeler job

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tweak rules

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-25 15:55:01 +00:00
tison
417be13400 feat: adopt human-panic crash reports (#2996) 
* feat: adopt human-panic crash reports

Signed-off-by: tison <wander4096@gmail.com>

* build: fix dep toml conflict

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2023-12-25 13:23:18 +00:00
Ruihang Xia
0a9ad004a4 chore: bump version to 0.5.0 (#3000) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-25 13:14:59 +00:00
SSebo
cf561df854 feat: export runtime metric to promethues (#2985) 
* feat: export runtime metric to promethues

* fix: export tokio metric should enable tokio_unstable

* chore: tokio-metric export add more check info
 2023-12-25 12:47:22 +00:00
LFC
1641fd572a refactor: hide RegionRoute behind TableRouteValue (#2989) 
* refactor: hide `RegionRoute` behind `TableRouteValue` (Metric Engine table route, part 2)

* Update src/common/meta/src/ddl/create_table.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/meta-srv/src/procedure/region_migration/test_util.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: resolve PR comments

* fix: rustfmt

* compatible with the old TableRouteValue

* Update src/common/meta/src/key/table_route.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2023-12-25 11:37:50 +00:00
Weny Xu
89129c99c8 chore: setup kafka standalone in coverage test (#2984) 
* chore: setup kafka standalone in coverage test

* test: add a naive test for topic manager
 2023-12-25 11:18:54 +00:00
Ruihang Xia
48cd22d459 feat: support querying metric engine from frontend (#2987) 
* query one logical table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* map column id

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove deadcode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove redundent column name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-25 10:47:57 +00:00
zyy17
0d42651047 feat: add '--store-key-prefix' to support multiple meta instances can use the same etcd backend (#2988) 
* feat: add '--store-key-prefix' to support multiple meta instances can use the same etcd backend

* refactor: refine lock_key
 2023-12-25 09:33:25 +00:00
niebayes
bab198ae68 feat(remote_wal): impl kafka log store (#2971) 
* feat: introduce client manager

* chore: add errors for client manager

* chore: add record utils

* chore: impl kafka log store

* chore: build kafka log store upon starting datanode

* chore: update comments for kafka log store

* chore: add a todo for getting entry offset

* fix: typo

* chore: remove unused

* chore: update comments

* fix: typo

* fix: resolve some review conversations

* chore: move commonly referenced crates to workspace Cargo.toml

* fix: style

* fix: style

* chore: unify topic name prefix

* chore: make backoff config configurable by users

* chore: properly use backoff config in wal config

* refactor: read/write of kafka log store

* fix: typo

* fix: typo

* fix: resolve review conversations
 2023-12-25 09:21:52 +00:00
niebayes
d4ac8734bc refactor(remote_wal): entry id usage (#2986) 
* chore: update comments for log store stuff

* refactor: entry id usage

* tmp update

* Revert "tmp update"

This reverts commit fcfcda2851.

* fix: resolve review conversations

* fix: resolve review conversations

* chore: remove entry offset
 2023-12-25 07:30:27 +00:00
Lei, HUANG
4664cc601c fix: install nextest bin (#2990) 
* use binstall to install nextest

* fix: change all docker files

* Update docker/dev-builder/centos/Dockerfile

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-25 06:01:47 +00:00
Weny Xu
06fd7fd210 feat: add skip_wal_replay to OpenRegion instruction (#2977) 
feat: add skip_wal_replay to OpenRegion instruction
 2023-12-23 06:42:21 +00:00
dennis zhuang
d7b2e791b9 fix: duplicate information_schema (#2979) 
* fix: duplicate information_schema

* chore: style

* fix: comment in sqlness
 2023-12-22 14:02:11 +00:00
niebayes
7d509e97f6 chore: move some commonly referenced crates to workspace Cargo.toml (#2981) 
fix: resolve conflicts
 2023-12-22 09:13:18 +00:00
Wei
a7349b573b feat: support fetch ranges in concurrent (#2959) 
* feat: concurrent fetch ranges

* chore: cr comment

* chore: cr comment

* chore: clippy
 2023-12-22 08:47:55 +00:00
niebayes
830a91c548 feat(remote_wal): implement topic allocation (#2970) 
* chore: implement wal options allocator

* chore: implement round-robin topic selector

* feat: add shuffle to round-robin topic selector

* chore: implement kafka topic manager

* test: add tests for wal options allocator

* test: add wal provider to test config files

* test: leave todos for adding tests for remote wal

* fix: resolve review conversations

* fix: typo
 2023-12-22 08:26:48 +00:00
ZonaHe
6221e5b105 feat: update dashboard to v0.4.4 (#2978) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2023-12-22 06:37:39 +00:00
Ning Sun
43f01cc594 feat: add a default internal schema (#2974) 2023-12-22 06:25:19 +00:00
WU Jingdi
675767c023 feat: Support export metric in remote write format (#2928) 
* feat: remote write metric task

* chore: pass stanalone task to frontend

* chore: change name to system metric

* fix: add header and rename to export metrics
 2023-12-22 02:09:12 +00:00
dennis zhuang
054bca359e feat: adds build_info table (#2969) 2023-12-21 12:28:09 +00:00
Weny Xu
ff8c10eae7 feat: add CatchupRequest to engine (#2939) 
* chore: remove redundant code

* feat(mito): add CatchupRequest

feat: reopen region before replay if need

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-12-21 08:27:53 +00:00
Zhenchi
b5c5458798 refactor: remove unused code for pruning row groups (#2973) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-12-21 07:54:29 +00:00
Ruihang Xia
6a1f5751c6 feat(metric-engine): open and close metric regions (#2961) 
* implementation

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add metrics

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* define consts

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix compile error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* print region id with display

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* only ignore region not found

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-21 07:33:22 +00:00
LFC
8776b1204b feat: use explicitly set table id (#2945) 
* feat: make standalone table metadata allocator able to use explicitly set table id

* rebase

* fix: resolve PR comments
 2023-12-21 05:51:43 +00:00
Yiran
bad89185c2 ci: add user-doc label checker (#2967) 2023-12-21 03:04:03 +00:00
Weny Xu
6c1c7d8d24 feat: allow initializing regions in background (#2930) 
* refactor: open regions in background

* feat: add status code of RegionNotReady

* feat: use RegionNotReady instead of RegionNotFound for a registering region

* chore: apply suggestions from CR

* feat: add status code of RegionBusy

* feat: return RegionBusy for mutually exclusive operations
 2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9 feat(remote_wal): add skeleton for remote wal related to datanode (#2941) 
* refactor: refactor wal config

* test: update tests related to wal

* feat: introduce kafka wal config

* chore: augment proto with wal options

* feat: augment region open request with wal options

* feat: augment mito region with wal options

* feat: augment region create request with wal options

* refactor: refactor log store trait

* feat: add skeleton for kafka log store

* feat: generalize building log store when starting datanode

* feat: integrate wal options to region write

* chore: minor update

* refactor: remove wal options from region create/open requests

* fix: compliation issues

* chore: insert wal options into region options upon initializing region server

* chore: integrate wal options into region options

* chore: fill in kafka wal config

* chore: reuse namespaces while writing to wal

* chore: minor update

* chore: fetch wal options from region while handling truncate/flush

* fix: region options test

* fix: resolve some review conversations

* refactor: serde with wal options

* fix: resolve some review conversations
 2023-12-20 09:01:17 +00:00
LFC
6ac47e939c feat: MetricsEngine table route (part 1) (#2952) 
* refactor: make `TableRouteValue` an enum to add the variant of MetricsEngine table route

* fix: resolve PR comments

* Update src/common/meta/src/key/table_route.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2023-12-20 07:31:59 +00:00
Zhenchi
7d1724f832 feat(inverted_index.create): add index creator (#2960) 
* feat(inverted_index.create): add read/write for external intermediate files

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: fix typos intermedia -> intermediate

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typos

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(inverted_index.create): add external sorter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: fix typos intermedia -> intermediate

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: drop the stream as early as possible to avoid recursive calls to poll

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: project merge sorted stream

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add total_row_count to SortOutput

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: remove change of format

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(inverted_index.create): add index creator

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add check for total_row_count

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: lazy set meta of writer

This reverts commit 63cb5bdb5c3a08406d978357d8167ca18ed1b83b.

* feat: lazyily provide inverted index writer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish readability

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add push_with_name_n

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-12-20 07:02:13 +00:00
Weny Xu
d2f49cbc2e chore: remove debug = 1 of dev profile (#2966) 2023-12-20 04:37:46 +00:00
Weny Xu
97c3755ab6 feat(mito): add skip_wal_replay option to OpenRegionRequest (#2955) 
* feat(mito): add skip_replay_wal option to OpenRegionRequest

* test: add tests for skip replay wal

* chore: rename `skip_replay_wal` to `skip_wal_replay`
 2023-12-20 03:31:48 +00:00
JeremyHi
5f8c17514f chore: SelectorType use snake_case (#2962) 
chore: let SelectorType use snake_case
 2023-12-20 02:28:29 +00:00
niebayes
839e653e0d feat(remote_wal): add skeleton for remote wal related to meta srv (#2933) 
* feat: introduce wal config and kafka config

* feat: introduce kafka topic manager and selector

* feat: introduce region wal options

* chore: build region wal options upon starting meta srv

* feat: integrate region wal options allocator into table meta allocator

* chore: add wal config to metasrv.example.toml

* chore: add region wal options map to create table procedure

* feat: augment region create request with wal options

* feat: augment DatanodeTableValue with region wal options map

* chore: encode region wal options upon constructing table creator

* feat: persist region wal options when creating table meta

* fix: sqlness test

* chore: set default wal provider to raft-engine

* refactor: refactor wal options

* chore: update wal options allocator

* refactor: rename region wal options to wal options

* chore: update usages of region wal options

* chore: add some comments to kafka

* chore: fill in kafka config

* test: add tests for serde wal config

* test: add tests for wal options

* refactor: refactor wal options allocator to enum

* refactor: store wal options into the request options instead

* fix: typo

* fix: typo

* refactor: move wal options map to region info

* refactor: refacto serialization and deserialization of wal options

* refactor: use serde_json to encode wal options

* chore: rename wal_options_map to region_wal_options

* chore: resolve some review comments

* fix: typo

* refactor: replace kecab-case with snake_case

* fix: sqlness and converage tests

* fix: typo

* fix: coverage test

* fix: coverage test

* chore: resolve some review conversations

* fix: resolve some review conversations

* chore: format comments in metasrv.example.toml

* chore: update import style

* feat: integrate wal options allocator to standalone mode

* test: add compatible test for OpenRegion

* test: add compatible test for UpdateRegionMetadata

* chore: remove based suffix from topic selector type
 2023-12-19 12:43:47 +00:00
LFC
c7b36779c1 fix: correctly decode mysql timestamp binary value (#2946) 
* fix: correctly decode mysql timestamp binary value

* fix: ut

* fix: resolve PR comments
 2023-12-19 11:05:28 +00:00
Wei
bbcac3a541 fix: auto create datatype_extension missing (#2953) 2023-12-19 10:53:31 +00:00
dennis zhuang
600cde1ff2 fix: wrong link for selector (#2958) 2023-12-19 10:50:48 +00:00
Zhenchi
83de399bef feat(inverted_index.create): add external sorter (#2950) 
* feat(inverted_index.create): add read/write for external intermediate files

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: MAGIC_CODEC_V1 -> CODEC_V1_MAGIC

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: fix typos intermedia -> intermediate

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typos

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(inverted_index.create): add external sorter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: fix typos intermedia -> intermediate

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: drop the stream as early as possible to avoid recursive calls to poll

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: project merge sorted stream

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add total_row_count to SortOutput

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: remove change of format

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: rename segment null bitmap

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: test type alias

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: allow `memory_usage_threshold` to be None to turn off dumping

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: change segment_row_count type to NonZeroUsize

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: accept BytesRef instead

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add `push_n` to adapt mito2

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add k-way merge TODO

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: more sorter cases

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: make the merge tree balance

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/index/src/inverted_index/create/sort/external_sort.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: stable feature

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-12-19 08:14:37 +00:00
Ruihang Xia
6b8dbcfb54 chore: update toolchain to 20231219 (#2932) 
* update toolchain file, remove unused feature gates

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update action file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update to 12-19

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-12-19 07:24:08 +00:00
Wei
3e6a564f8e fix: blocking read timer lack of parameter (#2954) 2023-12-19 06:16:55 +00:00
Zhenchi
ccbd49777d fix: correct the error message for snappy compress (#2956) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-12-19 06:14:40 +00:00
Weny Xu
29fc2ea9d8 feat: fetch manifests in concurrent (#2951) 
* feat: fetch manifests in concurrent

* chore: set fetching manifest concurrency limit to 16
 2023-12-19 06:12:30 +00:00
WU Jingdi
d180e41230 fix: change range query time slot to [align_ts, align_ts + range) (#2938) 2023-12-19 02:35:14 +00:00
liyang
62d5fcbd76 refactor: greptimedb cluster sqlness test scripts (#2947) 2023-12-18 10:59:14 +00:00
Zhenchi
d339191e29 refactor: remove reduntant .compat() (#2949) 
* refactor: remove reduntant `.compat()`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove dep

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-12-18 10:36:45 +00:00