rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-07 22:02:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,930 Commits 103 Branches 226 Tags
cfe3a2c55ef1989b2d85e4ebc56c21b95bdf3f51
Commit Graph

127 Commits

Author SHA1 Message Date
Niwaka
cfe3a2c55e feat!: support table ddl for custom storage (#2733) 
* feat: support table ddl for custom_storage

* refactor: rename extract_variant_name to name

* chore: add blank

* chore: keep compatible

* feat: rename custom_stores to providers

* chore: rename

* chore: config

* refactor: add should_retry in client Error

* fix: test fail

* chore: remove unused options

* chore: remove unused import

* chore: remove the blanks.

* chore: revert

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2023-12-06 15:59:01 +00:00
Yingwen
abbac46c05 fix: do not expose manifest compression algorithm (#2835) 
* fix: don't allow to set manifest compression algorithm

* docs: update config examples
 2023-11-29 05:49:40 +00:00
Wei
3a4c9f2b45 feat: supports decimal type in RPC (#2788) 
* refactor: ColumnDataTypeWrapper

* feat: decimal128 grpc

* feat: add test case

* chore: add TODO

* chore: empty line

* chore: remove precision and scale

* refactor: remove precision and scale

* chore: remove sqlness test

* chore: rename

* chore: proto version

* chore: cr comment.

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/mito2/src/memtable/time_series.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-11-24 09:19:33 +00:00
Ning Sun
85eebcb16f fix: correct mysql timestamp encoding for binary protocol (#2797) 
* fix: correct mysql timestamp encoding for binary protocol

* test: add tests for mysql timestamp encoding
 2023-11-23 18:54:59 +00:00
tison
102e43aace test: use EtcdStore in IT cases (#2734) 
* test: use EtcdStore in IT cases

Signed-off-by: tison <wander4096@gmail.com>

* retrigger CI

Signed-off-by: tison <wander4096@gmail.com>

* refactor: KvPair can take etcd KeyValue

Signed-off-by: tison <wander4096@gmail.com>

* temporary use fork

Signed-off-by: tison <wander4096@gmail.com>

* drop cloned

Signed-off-by: tison <wander4096@gmail.com>

* chroot_key_value

Signed-off-by: tison <wander4096@gmail.com>

* chroot and prepend in each point

Signed-off-by: tison <wander4096@gmail.com>

* adjust call points

Signed-off-by: tison <wander4096@gmail.com>

* cargo clippy

Signed-off-by: tison <wander4096@gmail.com>

* point to upstream etcd-client

Signed-off-by: tison <wander4096@gmail.com>

* test etcd chroot

Signed-off-by: tison <wander4096@gmail.com>

* add NO_CHROOT constant

Signed-off-by: tison <wander4096@gmail.com>

* check

Signed-off-by: tison <wander4096@gmail.com>

* handle range end

Signed-off-by: tison <wander4096@gmail.com>

* handle special encoded key or range_end

Signed-off-by: tison <wander4096@gmail.com>

* fixup implementation

Signed-off-by: tison <wander4096@gmail.com>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* avoid test name conflict

Signed-off-by: tison <wander4096@gmail.com>

* chroot to kvbackend level

Signed-off-by: tison <wander4096@gmail.com>

* fixup all occurances

Signed-off-by: tison <wander4096@gmail.com>

* fix type

Signed-off-by: tison <wander4096@gmail.com>

* Update src/common/meta/src/kv_backend/txn.rs

* make github happy

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
 2023-11-23 05:47:00 +00:00
Yingwen
b9146c88ff refactor: Remove usages of the old storage crate (#2777) 
* chore: remove storage from some crate

* feat: remove storage config

* feat: remove storage from cmd

* feat: impl stream_to_parquet

* feat: remove storage from operator

* feat: remove stream writer from mito2

* feat: remove storage from project toml

* test: fix config api test

* docs: remove outdated configs

* refactor: remove storage directory
 2023-11-20 20:29:41 +08:00
fys
da68d8ce4b feat: add random weighted choose in load_based selector (#2234) 
* feat: add random weigted choose in load_based selector

* fix: meta cannot save heartbeats when cluster have no region

* chore: print some log

* chore: remove unused code

* cr

* add some logs when filter result is empty
 2023-11-20 06:47:42 +00:00
Yingwen
ce959ddd3f feat(mito): implements row group level page cache (#2688) 
* feat: add page cache

* docs: update mito config toml

* feat: impl CachedPageReader

* feat: use cache reader to read row group

* feat: do not fetch data if we have pages in cache

* chore: return if nothing to fetch

* feat: enlarge page cache size

* test: test write read parquet

* test: test cache

* docs: update comments

* test: fix config api test

* feat: cache metrics

* feat: change default page cache size

* test: fix config api test
 2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40 feat: Enable distributed tracing in greptimedb (#2755) 
* feat: implement distributed tracing

* fix: change usage of span

* fix: use otlp as exporter

* chore: update dependence

* chore: add span info

* chore: add alias

* chore: use instrument instead of trace
 2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074 fix!: improve user experience on setting compression type (#2765) 
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type

* Update src/common/datasource/src/compression.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-11-17 06:27:13 +00:00
Ruihang Xia
6599bb5a46 feat: do not require worker's number to be power of 2 (#2732) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-11-13 09:04:44 +00:00
tison
f5eede4ce1 feat: support prometheus format_query endpoint (#2731) 
* feat: support prometheus format_query endpoint

Signed-off-by: tison <wander4096@gmail.com>

* add test

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2023-11-10 10:33:28 +00:00
Lei, HUANG
b53537e69b fix: unify all sst_write_buffer_size usage (#2712) 
* fix: unify all sst_write_buffer_size usage

* fix: some CR comments

* fix: logs
 2023-11-09 11:22:47 +00:00
Niwaka
06d273b75a fix: GET: /config return all configurations when running standalone (#2630) 
* fix: config api return all configurations when running standalone

* chore: follow same style

* fix: avoid panic
 2023-11-08 03:22:19 +00:00
Weny Xu
fb8d0c6ce5 refactor: unify the KvBackend (#2684) 
* refactor: unify the KvBackend

* refactor: rename kv_store to kv_backend

* chore: apply suggestions from CR

* refactor: rename kv_store to kv_backend

* refactor: rename KvStoreConfig to KvBackendConfig
 2023-11-03 02:46:43 +00:00
WU Jingdi
180bc64cb0 feat: change metric crate from metrics to prometheus (#2655) 
* feat: change metrics to prometheus

* chore: fix code advise

* chore: resolve merge conflict

* chore: fix code advise
 2023-10-31 15:46:57 +00:00
Yun Chen
f859932745 fix: convert to ReadableSize & Durations (#2594) 
* fix: convert to ReadableSize & Durations

* fix: change more grpc sender/recv message size to ReadableSize

fix: format

fix: cargo fmt

fix: change cmd test to use durations

fix: revert metaclient change

fix: convert default fields in meta client options

fix: human serde meta client durations

* fix: remove milisecond postfix in heartbeat option

* fix: humantime serde on heartbeat

* fix: update config example

* fix: update integration test config

* fix: address pr comments

* fix: fix pr comment on default annotation
 2023-10-13 03:28:29 +00:00
dennis zhuang
ff15bc41d6 feat: improve object storage cache (#2522) 
* feat: refactor object storage cache with moka

* chore: minor fixes

* fix: concurrent issues and invalidate cache after write/delete

* chore: minor changes

* fix: cargo lock

* refactor: rename

* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib

* fix: typo

* chore: style

* fix: toml format

* chore: toml

* fix: toml format

* Update src/object-store/src/layers/lru_cache/read_cache.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: update src/object-store/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: refactor and apply suggestions

* fix: typo

* feat: adds back allow list for caching

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr suggestion

Co-authored-by: Yingwen <realevenyag@gmail.com>

* refactor: wrap inner Accessor with Arc

* chore: remove run_pending_task in read and write path

* chore: the arc is unnecessary

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-10-08 03:27:49 +00:00
Yingwen
657542c0b8 feat(mito): Cache repeated vector for tags (#2523) 
* feat: add vector_cache to CacheManager

* feat: cache repeated vectors

* feat: skip decoding pk if output doesn't contain tags

* test: add TestRegionMetadataBuilder

* test: test ProjectionMapper

* test: test vector cache

* test: test projection mapper convert

* style: fix clippy

* feat: do not cache vector if it is too large

* docs: update comment
 2023-10-07 11:36:00 +00:00
Ning Sun
0ad3fb6040 fix: mysql timezone settings (#2534) 
* fix: restore time zone settings for mysql

* test: add integration test for time zone

* test: fix unit test for check
 2023-10-07 10:21:32 +00:00
Ruihang Xia
e352fb4495 fix: check for table scan before expanding (#2491) 
* fix: check for table scan before expanding

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change assert_ok to unwrap

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't skip dml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* uncomment ignored tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-09-26 12:12:08 +00:00
Yingwen
a6116bb866 feat(mito): Add cache manager (#2488) 
* feat: add cache manager

* feat: add cache to reader builder

* feat: add AsyncFileReaderCache

* feat: Impl AsyncFileReaderCache

* chore: move moka dep to workspace

* feat: add moka cache to the manager

* feat: implement parquet meta cache

* test: test cache manager

* feat: consider vec size

* style: fix clippy

* test: fix config api test

* feat: divide cache

* test: test disabling meta cache

* test: fix config api test

* feat: remove meta cache if file is purged
 2023-09-26 11:46:19 +00:00
LinFeng
621c6f371b feat: limit grpc message size (#2459) 
* feat: add two grpc config options

Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size

* test: add integration tests for message size limit
 2023-09-22 02:07:46 +00:00
shuiyisong
20f4f7971a refactor: remove source and location in snafu display (#2428) 
* refactor: remove source pt 1

* refactor: remove source pt 2

* refactor: remove source pt 3

* refactor: remove location pt 1

* refactor: remove location pt 2

* chore: remove rustc files

* chore: fix error case

* chore: fix test

* chore: fix test

* chore: fix cr issue

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
 2023-09-21 02:55:24 +00:00
Ruihang Xia
34d6288945 feat: bring back sqlness and integration tests (#2448) 
* enable integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* disable sqlness region failover

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enable sqlness in CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort unstable result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* set require_lease_before_startup to true

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix inconsistent cache

* replace windows path chars

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore some integration cases in windows

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "ignore some integration cases in windows"

This reverts commit 122478b7c1.

* disable windows for now

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix close region bug in RegionHeartbeatResponseHandler

* disable failover tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-20 09:17:30 +00:00
dennis zhuang
342cc0a4c4 fix: compile error after updating protos (#2435) 2023-09-18 12:12:39 +00:00
Zhenchi
d6d46378a1 test: fix some integration tests (#2432) 
* test: fix some integration tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: add timezone setting

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2023-09-18 10:52:14 +00:00
Zhenchi
3a39215f11 feat: migrate file engine from table to reigon (#2365) 
* feat: migrate file engine from table to reigon

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update src/file-engine/src/engine.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: specify ts index for file engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: handle time index for external table

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: some integration testsg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add file schema and table schema compatibility

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: compatible file schema to region schema

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: add error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: simplify close

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: implement set_writable

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: tests-integration compilation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local>
 2023-09-18 08:02:43 +00:00
LFC
4b13c88752 fix: resolve more integration tests (#2406) 
* fix: resolve more integration tests

* Update tests-integration/tests/http.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2023-09-15 09:43:16 +00:00
dennis zhuang
43e3c94fd1 refactor: catalog managers (#2405) 
* feat: rename catalog::local to catalog::memory

* refactor: catalog managers

* chore: license header
 2023-09-15 08:48:14 +00:00
Yingwen
a84a8ad04f fix: alter table procedure panics while renaming table (#2397) 
* fix: procedure panic on renaming table

* test: fix test_insert_and_select invalid arguments

* test: fix test_standalone_insert_and_query using wrong semantic type

* test: fix test_distributed_insert_delete_and_query semantic type
 2023-09-14 11:50:00 +00:00
LFC
d08b05c963 fix: make test-integration able to compile (#2384) 
* fix: make test-integration able to compile

* chore: fmt toml

---------

Co-authored-by: WenyXu <wenymedia@gmail.com>
 2023-09-13 12:42:46 +00:00
JeremyHi
58d07e0e62 feat: v04 rm unused exprs (#2285) 
* feat: rm compact and flush exprs

* refactor: continue to rm compact and flush
 2023-09-12 07:57:15 -05:00
shuiyisong
f16f58266e refactor: query_ctx from http middleware (#2253) 
* chore: change userinfo to query_ctx in http handler

* chore: minor change

* chore: move prometheus http to http mod

* chore: fix uni test:

* chore: add back schema check

* chore: minor change

* chore: remove clone
 2023-08-25 09:36:33 +00:00
Ning Sun
88247e4284 fix!: resolve residual issues with removing prometheus port (#2227) 
* fix: resolve residual issues when removing prometheus port

* fix: remove prometheus from sample config as well
 2023-08-23 01:49:11 +00:00
dennis zhuang
18fa0e01ed feat: remove checkpoint_on_startup (#2228) 
feat: update flushed manifest version when it is larger
 2023-08-22 13:09:34 +00:00
niebayes
e5f4ca2dab feat: streaming do_get (#2171) 
* feat: rewrite do_get for streaming get flight data

* feat: rewrite do_get call stack but leave the async stream adapter not modified yet

* feat: rewrite the async stream adapter to accept greptime record batch stream

* fix: resolve some PR comments

* feat: rewrite tests to adapt to the streaming do_get

* feat: add unit tests for streaming do_get

* feat: rewrite timer metric of merge scan

* remove unhelpful unit tests for streaming do_get

* add a new metric timer for merge scan and fix some test errors

* rewrite mysql writer to write query results in a streaming manner

* fix: fix fmt errors

* fix: rewrite sqlness runner to take into account the streaming do_get

* fix: fix toml format errors

* fix: resolve some PR comments

* fix: resolve some PR comments

* fix: refactor do_get to increase readability

* fix: refactor mysql try_write_one to increase readability
 2023-08-22 02:54:05 +00:00
Weny Xu
e1ce1d86a1 refactor: unite key serialization method (#2195) 2023-08-18 09:42:19 +00:00
Weny Xu
6e6ff5a606 refactor: update table metadata in single txn (#2172) 
* refactor: table-metadata-manager

* feat: remove comparing when deleting metadata

* fix: fix comment typos

* chore: apply suggestions from CR

* test: add tests for updating DatanodeTable

* fix: fix clippy

* chore: apply suggestions from CR

* refactor: improve update table route tests

* refactor: return Txn instead of TxnRequest

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: update table metadata in single txn

* feat: check table exists before drop table executing

* test: add tests for table metadata manager

* refactor: remove table region manager

* chore: apply suggestions from CR

* feat: add bench program

* chore: apply suggestions from CR
 2023-08-17 06:29:19 +00:00
sh2
202540823f refactor!: move prometheus routes to default http server (#2005) 
* move prometheus routes to default http server

Signed-off-by: sh2 <shawnhxh@outlook.com>

* fix ci test and remove the server logic of prometheus

* remove unused import and prometheus relevant code

* fix ci: rustfmt and test

* fix ci: silly fmt

* fix ci: silly silly fmt

* change `/prom_store` back to `/prometheus`

* remove unused variable

---------

Signed-off-by: sh2 <shawnhxh@outlook.com>
 2023-08-16 03:21:14 +00:00
shuiyisong
7f51141ed0 refactor: auth crate (#2148) 
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
 2023-08-14 02:51:26 +00:00
dennis zhuang
6f40128058 feat!: enable telemetry by default (#2137) 
* feat: remove greptimedb-telemetry feature

* feat: adds enable_telemetry option to metasrv and datanode

* refactor: move data_home from file config to storage config

* feat: store the installation uuid into datanode and metasrv working home

* fix: cargo toml fmt

* test: ignore region failver test when using local fle storage

* test: ignore telemetry reporter in test mode

* feat: print warning log when enabling telemetry

* chore: the telemetry doc link

* chore: remove enable_telemetry from datanode example config file

* refactor: rename GREPTIMEDB_TELEMETRY_CLIENT_REQUEST_TIMEOUT

* chore: rename print_warn_log to print_anonymous_usage_data_disclaimer
 2023-08-11 14:50:40 +00:00
LFC
5c19913a91 build: on windows (#2054) 
* build on windows

* rebase develop

* fix: resolve PR comments
 2023-08-10 08:08:37 +00:00
Ning Sun
d18eb18b32 feat: use server inferenced types on statement describe (#2032) 
* feat: use server inferenced types on statement describe

* feat: add support for server inferenced type

* feat: allow parameter type inferencing

* chore: update comments

* fix: lint issue

* style: comfort rustfmt

* Update src/servers/src/postgres/types.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-09 02:57:56 +00:00
Ruihang Xia
5f0403c245 feat: improve /label and /labels APIs in prometheus server (#2087) 
* support __name__ for /label

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* make match[] in labels optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-03 07:51:08 +00:00
shuiyisong
5eb2c609a3 fix: auth in grpc (#2056) 
* fix: auth in grpc

* fix: change to return err

* fix: add grpc test

* fix: add http test

* fix: add mysql and pg test
 2023-08-01 15:18:31 +00:00
JeremyHi
fd7eb87a52 refactor: common semantic-type (#2057) 2023-08-01 11:18:05 +00:00
JeremyHi
41139ec11d feat: region lease improve (#2004) 
* feat: add exists api into KvBackend

* refactor: region lease

* feat: fiter out inactive node in keep-lease

* feat: register&deregister inactive node

* chore: doc

* chore: ut

* chore: minor refactor

* feat: use memory_kv to store inactive node

* fix: use real error in

* chore: make inactive_node_manager's func compact

* chore: more efficiently

* feat: clear inactive status on cadidate node
 2023-07-24 03:49:14 +00:00
LFC
172febb1af refactor!: trying to replace TableGlobalValue, part 2 (#1985) 
* refactor!: using the new table metadata values

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-07-19 12:01:43 +00:00
Eugene Tolbakov
6811acb314 fix(status_endpoint): add default value for hostname (#1972) 
* fix(status_endpoint): add default value for hostname

* fix: adjust according to clippy suggestions

* fix: adjust according to CR suggestions
 2023-07-17 11:55:11 +00:00