rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-09 06:42:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,875 Commits 104 Branches 226 Tags
56fc77e573517fc3263f2237a2edc196ae383da1
Commit Graph

550 Commits

Author SHA1 Message Date
tison
efc5abfc02 build: upgrade etcd-client to 0.12.2 (#2781) 
* build: upgrade etcd-client to 0.12.2

Signed-off-by: tison <wander4096@gmail.com>

* upgrade nightly toolchain

Signed-off-by: tison <wander4096@gmail.com>

* chore: run clippy

Signed-off-by: tison <wander4096@gmail.com>

* Update Cargo.toml

Co-authored-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
 2023-11-21 06:33:41 +00:00
Ruihang Xia
20f01219e9 refactor: adjust metric engine structure (#2773) 
* extract consts into a separate mod

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add documents

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* split state

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* split create

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* split alter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* split put

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix metadata clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix engine clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-21 05:21:22 +00:00
Yingwen
b9146c88ff refactor: Remove usages of the old storage crate (#2777) 
* chore: remove storage from some crate

* feat: remove storage config

* feat: remove storage from cmd

* feat: impl stream_to_parquet

* feat: remove storage from operator

* feat: remove stream writer from mito2

* feat: remove storage from project toml

* test: fix config api test

* docs: remove outdated configs

* refactor: remove storage directory
 2023-11-20 20:29:41 +08:00
Bruce Chen
9558b3c201 build(cmd): upgrade clap to 4.x (#2775) 
* build(cmd): upgrade clap to 4.4.8

* build(cmd): upgrade clap to 4.4
 2023-11-20 10:43:31 +00:00
fys
da68d8ce4b feat: add random weighted choose in load_based selector (#2234) 
* feat: add random weigted choose in load_based selector

* fix: meta cannot save heartbeats when cluster have no region

* chore: print some log

* chore: remove unused code

* cr

* add some logs when filter result is empty
 2023-11-20 06:47:42 +00:00
Zhenchi
d9eeeee06e feat(puffin): add file reader (#2751) 
* feat(puffin): add file reader

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: toml format

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename PuffinParser to PuffinFileReader

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/puffin/src/file_format/reader/footer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/puffin/src/file_format/reader/file.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/puffin/src/file_format/reader/footer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/puffin/src/file_format/reader/footer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: check file size

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: redundant type cast

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: reuse read buffer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: check payload size

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: check payload size

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: validate blob offset

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: validate blob offset

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-11-20 04:29:41 +00:00
WU Jingdi
500e299e40 feat: Enable distributed tracing in greptimedb (#2755) 
* feat: implement distributed tracing

* fix: change usage of span

* fix: use otlp as exporter

* chore: update dependence

* chore: add span info

* chore: add alias

* chore: use instrument instead of trace
 2023-11-17 08:51:57 +00:00
Ruihang Xia
ac4b6cd7f0 feat: write logical region to metric engine (#2759) 
* transform write request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add tests for put request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use table_id instead of metric_name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* CR sugg.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* define random state as const

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-17 07:44:11 +00:00
Lei, HUANG
e083b8011c chore: bump greptimedb version v0.4.3 (#2766) 2023-11-17 06:12:37 +00:00
Yingwen
142035340d fix: compiler errors under non-default features (#2761) 
* feat: bump prost and fix pprof feature compiler errors

* feat: fix compiler errors on tokio-console

* chore: fix compiler errors

* ci: add all features check to ci
 2023-11-16 19:43:44 +00:00
Wei
ae27fbc7f2 feat: impl Vector, DataType traits for decimal128 (#2727) 
* feat: decimal128 vector

* feat: Decimal128Type in datatypes

* refactor: cast function

* feat: decimal128 type in ConcreteDataType

* chore: decimal

* chore: sql_data_type_to_concrete_data_type

* chore: cr comment

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: fix finish()

* chore: add debug_assert to new decimal128

---------

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-11-16 06:44:26 +00:00
Zhenchi
3329da5b72 feat(puffin): add partial reader (#2741) 
* feat(puffin): add partial reader

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-11-15 06:28:20 +00:00
Ruihang Xia
5b978269cc feat: simplify log file output by removing bunyan layer (#2747) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-14 14:18:34 +00:00
LFC
3dffc7b62c fix: upgrade dependency to resolve github security alert (#2744) 2023-11-14 09:11:43 +00:00
Zhenchi
1757061272 feat: introduce metadata struct of puffin (#2725) 
* feat: introduce metadata struct of puffin

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address suggestions

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-11-13 10:02:40 +00:00
Ruihang Xia
6599bb5a46 feat: do not require worker's number to be power of 2 (#2732) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-13 09:04:44 +00:00
tison
af7107565a refactor: use humantime for user-facing options (#2724) 
* refactor: use humantime for user-facing options

Signed-off-by: tison <wander4096@gmail.com>

* unify quote style

Signed-off-by: tison <wander4096@gmail.com>

* taplo fmt

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2023-11-09 13:29:47 +00:00
Weny Xu
a3611516a2 refactor: heartbeat response contains region role (#2718) 2023-11-09 09:24:36 +00:00
Ruihang Xia
93f21b188d chore: unify workspace dependencies and remove some unused (#2713) 
* log-store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* base64

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* udeps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change the workspace deps' style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-09 07:34:22 +00:00
Ruihang Xia
c62ba79759 feat: create logical region to metric engine (#2706) 
* wip: create logical region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change metadata region to handle table id rather than table name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement create logic region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add some tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement alter logical region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test, style and document

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't change default constraint

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-09 06:04:04 +00:00
Niwaka
06d273b75a fix: GET: /config return all configurations when running standalone (#2630) 
* fix: config api return all configurations when running standalone

* chore: follow same style

* fix: avoid panic
 2023-11-08 03:22:19 +00:00
Wei
0a91335e24 feat: decimal128 type in common (#2696) 
* feat: decimal type in common

* chore: rename

* feat: decimal128 compare

* chore: clippy

* chore: typos

* chore: cr comment

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>

* chore: cr comment.

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-11-06 08:04:50 +00:00
Ruihang Xia
6fd04e38a3 feat: implement create region request for metric engine (#2694) 
* implement basic put/get/exist interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test add_column and add_table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move engine test to test_util

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* verify incoming create region request

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* create data region

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply review sugg

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-06 03:48:26 +00:00
WU Jingdi
bbaae9223a feat: enable empty range by (#2697) 2023-11-06 03:45:33 +00:00
Weny Xu
060864d0c1 feat: decrease the page size if the response size exceeds the limit (#2689) 
* feat: decrease the `page size` if the response message size exceeds the limit

* chore: apply suggestions from CR

* feat: prefer to use adaptive_page_size

* chore: apply suggestions from CR
 2023-11-06 03:36:15 +00:00
Ruihang Xia
0dca63bc7b feat: Metric engine skeleton (#2687) 
* metadata region logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix mito2 test feature gate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* override unused warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add basic test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply review sugg

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-03 08:34:08 +00:00
zyy17
7bd137f398 chore: update greptime-proto to main (#2672) 2023-11-01 03:13:36 +00:00
WU Jingdi
180bc64cb0 feat: change metric crate from metrics to prometheus (#2655) 
* feat: change metrics to prometheus

* chore: fix code advise

* chore: resolve merge conflict

* chore: fix code advise
 2023-10-31 15:46:57 +00:00
Weny Xu
54ed7529ca feat: introduce the region role (#2640) 
* feat: introduce region role

* chore: apply suggestions from CR
 2023-10-31 06:59:22 +00:00
Weny Xu
88eb69530a refactor: remove router rpc (#2646) 2023-10-31 04:54:56 +00:00
Weny Xu
bd177b8cc4 refactor: remove move_value (#2661) 
* chore: bump orc-rust to 0.2.42

* refactor: remove move_value
 2023-10-27 03:42:50 +00:00
Ruihang Xia
5d8b0e8154 chore: bump dependencies (#2659) 
* fix all hard error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix nextest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* trivial changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix order by

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sql keyword and data type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix range exec's input partitioning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix cover input type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix explain analyze

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update distributed mode sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix lints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update locks

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* downgrade dlv-list

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-26 12:24:14 +00:00
liyang
01fdbf3626 chore: upgrade 0.4.2 (#2644) 2023-10-24 12:21:58 +08:00
Niwaka
9dcfd28f61 feat: impl ObjectStoreManager for custom_storage (#2621) 
* feat: impl ObjectStoreManager for custom_storage

* fix: rename object_store_manager to manager

* fix: rename global to default

* chore: add document for ObjectStoreManager

* refactor: simplify default_object_store

* fix: address review
 2023-10-23 03:00:29 +00:00
Ruihang Xia
212ea2c25c feat: implement HistogramFold plan for prometheus histogram type (#2626) 
* basic impl of fold plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add schema test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fill plan attributes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* unify variable names

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-20 07:42:10 +00:00
WU Jingdi
46e106bcc3 feat: allow nest range expr in Range Query (#2557) 
* feat: eable range expr nest

* fix: change range expr rewrite format

* chore: organize range query tests

* chore: change range expr name(e.g. MAX(v) RANGE 5s FILL 6)

* chore: add range query test

* chore: fix code advice

* chore: fix ca
 2023-10-18 07:03:26 +00:00
zyy17
3217b56cc1 ci: release new version '0.4.0' -> '0.4.1' (#2611) 2023-10-17 07:33:41 +00:00
Yingwen
8a0054aa89 fix: make nyc-taxi bench work again (#2599) 
* fix: invalid requests created by nyc-taxi

* feat: add timestamp to table name

* style: fix clippy

* chore: re-export deps for client

* fix: wait result

* chore: no need to define a prefix constant
 2023-10-13 08:16:26 +00:00
Yun Chen
f859932745 fix: convert to ReadableSize & Durations (#2594) 
* fix: convert to ReadableSize & Durations

* fix: change more grpc sender/recv message size to ReadableSize

fix: format

fix: cargo fmt

fix: change cmd test to use durations

fix: revert metaclient change

fix: convert default fields in meta client options

fix: human serde meta client durations

* fix: remove milisecond postfix in heartbeat option

* fix: humantime serde on heartbeat

* fix: update config example

* fix: update integration test config

* fix: address pr comments

* fix: fix pr comment on default annotation
 2023-10-13 03:28:29 +00:00
Yingwen
7f75190fce chore: update Cargo.lock (#2564) 2023-10-10 16:28:50 +08:00
Ruihang Xia
07be50403e feat: add basic metrics to query (#2559) 
* add metrics to merge scan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* count series in promql

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tweak label name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* tweak label name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* document metric label

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-10 06:55:25 +00:00
Lei, HUANG
8bdef9a348 feat: memtable filter push down (#2539) 
* feat: memtable support filter pushdown to prune primary keys

* fix: switch to next time series when pk not selected

* fix: allow predicate evaluation failure

* fix: some clippy warnings

* fix: panic when no primary key in schema

* feat: cache decoded record batch for primary key

* refactor: use arcswap instead of rwlock

* fix: format toml
 2023-10-10 04:03:10 +00:00
Yingwen
d4577e7372 feat(mito): add metrics to mito engine (#2556) 
* feat: allow discarding a timer

* feat: flush metrics

* feat: flush bytes and region count metrics

* refactor: add as_str to get static string

* feat: add handle request elapsed metrics

* feat: add some write related metrics

* style: fix clippy
 2023-10-10 03:53:17 +00:00
Weny Xu
81ccb58fb4 refactor!: compare with origin bytes during the transactions (#2538) 
* refactor: compare with origin bytes during the transaction

* refactor: use serialize_str instead

* Update src/common/meta/src/key.rs

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* chore: apply suggestions from CR

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-10-09 08:17:19 +00:00
Weny Xu
ce3c10a86e refactor: de/encode protobuf-encoded byte array with base64 (#2545) 2023-10-09 05:31:44 +00:00
shuiyisong
007f7ba03c refactor: extract plugins crate (#2487) 
* chore: move frontend plugins fn

* chore: move datanode plugins to fn

* chore: add opt plugins

* chore: add plugins to meta-srv

* chore: setup meta plugins, wait for router extension

* chore: try use configurator for grpc too

* chore: minor fix fmt

* chore: minor fix fmt

* chore: add start meta_srv for hook

* chore: merge develop

* chore: minor fix

* chore: replace Arc<Plugins> with PluginsRef

* chore: fix header

* chore: remove empty file

* chore: modify comments

* chore: remove PluginsRef type alias

* chore: remove `OptPlugins`
 2023-10-09 04:54:27 +00:00
Ruihang Xia
d5e4fcaaff feat: dist plan optimize part 2 (#2543) 
* allow udf and scalar fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* put CountWildcardRule before dist planner

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* bump datafusion to fix first_value/last_value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use retain instead

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-10-09 02:18:36 +00:00
shuiyisong
067917845f fix: carry dbname from frontend to datanode (#2520) 
* chore: add dbname in region request header for tracking purpose

* chore: fix handle read

* chore: add write meter

* chore: add meter-core to dep

* chore: add converter between RegionRequestHeader and QueryContext & update proto version
 2023-10-08 06:30:23 +00:00
dennis zhuang
ff15bc41d6 feat: improve object storage cache (#2522) 
* feat: refactor object storage cache with moka

* chore: minor fixes

* fix: concurrent issues and invalidate cache after write/delete

* chore: minor changes

* fix: cargo lock

* refactor: rename

* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib

* fix: typo

* chore: style

* fix: toml format

* chore: toml

* fix: toml format

* Update src/object-store/src/layers/lru_cache/read_cache.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: update src/object-store/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: refactor and apply suggestions

* fix: typo

* feat: adds back allow list for caching

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* refactor: wrap inner Accessor with Arc

* chore: remove run_pending_task in read and write path

* chore: the arc is unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-10-08 03:27:49 +00:00
Yingwen
a6116bb866 feat(mito): Add cache manager (#2488) 
* feat: add cache manager

* feat: add cache to reader builder

* feat: add AsyncFileReaderCache

* feat: Impl AsyncFileReaderCache

* chore: move moka dep to workspace

* feat: add moka cache to the manager

* feat: implement parquet meta cache

* test: test cache manager

* feat: consider vec size

* style: fix clippy

* test: fix config api test

* feat: divide cache

* test: test disabling meta cache

* test: fix config api test

* feat: remove meta cache if file is purged
 2023-09-26 11:46:19 +00:00