67 Commits

Author SHA1 Message Date
LFC
196c06db14 feat: make logging to stdout configurable (#3003)
* feat: make logging to stdout configurable

* fix sqlness

* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380 refactor(remote_wal): add StandaloneWalConfig (#3002)
* feat: integrate remote wal to standalone

* fix: test

* refactor: refactor standalone wal config

* chore: change default kafka port to 9092

* chore: apply suggestions from CR

---------

Co-authored-by: niebayes <niebayes@gmail.com>
2023-12-26 04:31:41 +00:00
WU Jingdi
675767c023 feat: Support export metric in remote write format (#2928)
* feat: remote write metric task

* chore: pass stanalone task to frontend

* chore: change name to system metric

* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
Weny Xu
6c1c7d8d24 feat: allow initializing regions in background (#2930)
* refactor: open regions in background

* feat: add status code of RegionNotReady

* feat: use RegionNotReady instead of RegionNotFound for a registering region

* chore: apply suggestions from CR

* feat: add status code of RegionBusy

* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9 feat(remote_wal): add skeleton for remote wal related to datanode (#2941)
* refactor: refactor wal config

* test: update tests related to wal

* feat: introduce kafka wal config

* chore: augment proto with wal options

* feat: augment region open request with wal options

* feat: augment mito region with wal options

* feat: augment region create request with wal options

* refactor: refactor log store trait

* feat: add skeleton for kafka log store

* feat: generalize building log store when starting datanode

* feat: integrate wal options to region write

* chore: minor update

* refactor: remove wal options from region create/open requests

* fix: compliation issues

* chore: insert wal options into region options upon initializing region server

* chore: integrate wal options into region options

* chore: fill in kafka wal config

* chore: reuse namespaces while writing to wal

* chore: minor update

* chore: fetch wal options from region while handling truncate/flush

* fix: region options test

* fix: resolve some review conversations

* refactor: serde with wal options

* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
JeremyHi
99dda93f0e feat: sql with influxdb v1 result format (#2917)
* feat: sql with influxdb v1 result format

* chore: add unit tests

* feat: minor refactor

* chore: by comment

* chore; u128 to u64 since serde can't deser u128 in enum

* chore: by comment

* chore: apply suggestion

* chore: revert suggestion

* chore: try again

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-12-13 16:15:37 +00:00
Yingwen
6a57f4975e perf(mito): scan SSTs and memtables in parallel (#2852)
* feat: seq scan support parallelism

* feat: scan region by parallelism in config

* feat: enlarge channel size

* chore: parallel builder logs

* feat: use parallel reader accroding to source num

* chore: 128 channel size

* feat: add fetch cost metrics

* feat: add channel size to config

* feat: builder cost

* feat: logs

* feat: compiler error

* feat: fetch cost

* feat: convert cost

* chore: Revert "feat: logs"

This reverts commit 01e0df2c3a.

* chore: fix compiler errors

* feat: reduce channel size to 32

* chore: use workspace tokio-stream

* test: test scan in parallel

* chore: comment typos

* refactor: build all sources first

* test: test 0 parallelism

* feat: use parallel scan by default

* docs: update config example

* feat: log parallelism

* refactor: keep config in engine inner

* refactor: rename parallelism method

* docs: update docs

* test: fix config api test

* docs: update parallel scan comment

* feat: 0 for default parallelism
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e feat!: support table ddl for custom storage (#2733)
* feat: support table ddl for custom_storage

* refactor: rename extract_variant_name to name

* chore: add blank

* chore: keep compatible

* feat: rename custom_stores to providers

* chore: rename

* chore: config

* refactor: add should_retry in client Error

* fix: test fail

* chore: remove unused options

* chore: remove unused import

* chore: remove the blanks.

* chore: revert

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-12-06 15:59:01 +00:00
Yingwen
abbac46c05 fix: do not expose manifest compression algorithm (#2835)
* fix: don't allow to set manifest compression algorithm

* docs: update config examples
2023-11-29 05:49:40 +00:00
Yingwen
b9146c88ff refactor: Remove usages of the old storage crate (#2777)
* chore: remove storage from some crate

* feat: remove storage config

* feat: remove storage from cmd

* feat: impl stream_to_parquet

* feat: remove storage from operator

* feat: remove stream writer from mito2

* feat: remove storage from project toml

* test: fix config api test

* docs: remove outdated configs

* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
Yingwen
ce959ddd3f feat(mito): implements row group level page cache (#2688)
* feat: add page cache

* docs: update mito config toml

* feat: impl CachedPageReader

* feat: use cache reader to read row group

* feat: do not fetch data if we have pages in cache

* chore: return if nothing to fetch

* feat: enlarge page cache size

* test: test write read parquet

* test: test cache

* docs: update comments

* test: fix config api test

* feat: cache metrics

* feat: change default page cache size

* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40 feat: Enable distributed tracing in greptimedb (#2755)
* feat: implement distributed tracing

* fix: change usage of span

* fix: use otlp as exporter

* chore: update dependence

* chore: add span info

* chore: add alias

* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074 fix!: improve user experience on setting compression type (#2765)
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type

* Update src/common/datasource/src/compression.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2023-11-17 06:27:13 +00:00
Ruihang Xia
6599bb5a46 feat: do not require worker's number to be power of 2 (#2732)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-11-13 09:04:44 +00:00
tison
f5eede4ce1 feat: support prometheus format_query endpoint (#2731)
* feat: support prometheus format_query endpoint

Signed-off-by: tison <wander4096@gmail.com>

* add test

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2023-11-10 10:33:28 +00:00
Lei, HUANG
b53537e69b fix: unify all sst_write_buffer_size usage (#2712)
* fix: unify all sst_write_buffer_size usage

* fix: some CR comments

* fix: logs
2023-11-09 11:22:47 +00:00
Niwaka
06d273b75a fix: GET: /config return all configurations when running standalone (#2630)
* fix: config api return all configurations when running standalone

* chore: follow same style

* fix: avoid panic
2023-11-08 03:22:19 +00:00
WU Jingdi
180bc64cb0 feat: change metric crate from metrics to prometheus (#2655)
* feat: change metrics to prometheus

* chore: fix code advise

* chore: resolve merge conflict

* chore: fix code advise
2023-10-31 15:46:57 +00:00
Yun Chen
f859932745 fix: convert to ReadableSize & Durations (#2594)
* fix: convert to ReadableSize & Durations

* fix: change more grpc sender/recv message size to ReadableSize

fix: format

fix: cargo fmt

fix: change cmd test to use durations

fix: revert metaclient change

fix: convert default fields in meta client options

fix: human serde meta client durations

* fix: remove milisecond postfix in heartbeat option

* fix: humantime serde on heartbeat

* fix: update config example

* fix: update integration test config

* fix: address pr comments

* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
Yingwen
657542c0b8 feat(mito): Cache repeated vector for tags (#2523)
* feat: add vector_cache to CacheManager

* feat: cache repeated vectors

* feat: skip decoding pk if output doesn't contain tags

* test: add TestRegionMetadataBuilder

* test: test ProjectionMapper

* test: test vector cache

* test: test projection mapper convert

* style: fix clippy

* feat: do not cache vector if it is too large

* docs: update comment
2023-10-07 11:36:00 +00:00
Yingwen
a6116bb866 feat(mito): Add cache manager (#2488)
* feat: add cache manager

* feat: add cache to reader builder

* feat: add AsyncFileReaderCache

* feat: Impl AsyncFileReaderCache

* chore: move moka dep to workspace

* feat: add moka cache to the manager

* feat: implement parquet meta cache

* test: test cache manager

* feat: consider vec size

* style: fix clippy

* test: fix config api test

* feat: divide cache

* test: test disabling meta cache

* test: fix config api test

* feat: remove meta cache if file is purged
2023-09-26 11:46:19 +00:00
LinFeng
621c6f371b feat: limit grpc message size (#2459)
* feat: add two grpc config options

Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size

* test: add integration tests for message size limit
2023-09-22 02:07:46 +00:00
shuiyisong
20f4f7971a refactor: remove source and location in snafu display (#2428)
* refactor: remove source pt 1

* refactor: remove source pt 2

* refactor: remove source pt 3

* refactor: remove location pt 1

* refactor: remove location pt 2

* chore: remove rustc files

* chore: fix error case

* chore: fix test

* chore: fix test

* chore: fix cr issue

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
2023-09-21 02:55:24 +00:00
Ruihang Xia
34d6288945 feat: bring back sqlness and integration tests (#2448)
* enable integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* disable sqlness region failover

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* enable sqlness in CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort unstable result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* set require_lease_before_startup to true

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix inconsistent cache

* replace windows path chars

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore some integration cases in windows

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "ignore some integration cases in windows"

This reverts commit 122478b7c1.

* disable windows for now

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix close region bug in RegionHeartbeatResponseHandler

* disable failover tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
2023-09-20 09:17:30 +00:00
Zhenchi
d6d46378a1 test: fix some integration tests (#2432)
* test: fix some integration tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: add timezone setting

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
2023-09-18 10:52:14 +00:00
LFC
4b13c88752 fix: resolve more integration tests (#2406)
* fix: resolve more integration tests

* Update tests-integration/tests/http.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
2023-09-15 09:43:16 +00:00
shuiyisong
f16f58266e refactor: query_ctx from http middleware (#2253)
* chore: change userinfo to query_ctx in http handler

* chore: minor change

* chore: move prometheus http to http mod

* chore: fix uni test:

* chore: add back schema check

* chore: minor change

* chore: remove clone
2023-08-25 09:36:33 +00:00
Ning Sun
88247e4284 fix!: resolve residual issues with removing prometheus port (#2227)
* fix: resolve residual issues when removing prometheus port

* fix: remove prometheus from sample config as well
2023-08-23 01:49:11 +00:00
dennis zhuang
18fa0e01ed feat: remove checkpoint_on_startup (#2228)
feat: update flushed manifest version when it is larger
2023-08-22 13:09:34 +00:00
sh2
202540823f refactor!: move prometheus routes to default http server (#2005)
* move prometheus routes to default http server

Signed-off-by: sh2 <shawnhxh@outlook.com>

* fix ci test and remove the server logic of prometheus

* remove unused import and prometheus relevant code

* fix ci: rustfmt and test

* fix ci: silly fmt

* fix ci: silly silly fmt

* change `/prom_store` back to `/prometheus`

* remove unused variable

---------

Signed-off-by: sh2 <shawnhxh@outlook.com>
2023-08-16 03:21:14 +00:00
shuiyisong
7f51141ed0 refactor: auth crate (#2148)
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
2023-08-14 02:51:26 +00:00
dennis zhuang
6f40128058 feat!: enable telemetry by default (#2137)
* feat: remove greptimedb-telemetry feature

* feat: adds enable_telemetry option to metasrv and datanode

* refactor: move data_home from file config to storage config

* feat: store the installation uuid into datanode and metasrv working home

* fix: cargo toml fmt

* test: ignore region failver test when using local fle storage

* test: ignore telemetry reporter in test mode

* feat: print warning log when enabling telemetry

* chore: the telemetry doc link

* chore: remove enable_telemetry from datanode example config file

* refactor: rename GREPTIMEDB_TELEMETRY_CLIENT_REQUEST_TIMEOUT

* chore: rename print_warn_log to print_anonymous_usage_data_disclaimer
2023-08-11 14:50:40 +00:00
LFC
5c19913a91 build: on windows (#2054)
* build on windows

* rebase develop

* fix: resolve PR comments
2023-08-10 08:08:37 +00:00
Ruihang Xia
5f0403c245 feat: improve /label and /labels APIs in prometheus server (#2087)
* support __name__ for /label

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* make match[] in labels optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2023-08-03 07:51:08 +00:00
shuiyisong
5eb2c609a3 fix: auth in grpc (#2056)
* fix: auth in grpc

* fix: change to return err

* fix: add grpc test

* fix: add http test

* fix: add mysql and pg test
2023-08-01 15:18:31 +00:00
Eugene Tolbakov
6811acb314 fix(status_endpoint): add default value for hostname (#1972)
* fix(status_endpoint): add default value for hostname

* fix: adjust according to clippy suggestions

* fix: adjust according to CR suggestions
2023-07-17 11:55:11 +00:00
Kree0
8f71ac2172 refactor: move heartbeat configuration into an independent section (#1976)
refactor: move heartbeat configuration into an independent section in config file

* refactor: move heartbeat configuration into an independent section in config file

* feat: add HeartbeatOptions struct

* test: modify corresponding test case

* chore: modify corresponding example file
2023-07-17 11:29:02 +08:00
Eugene Tolbakov
674bfd85c7 chore(prom)!: rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus (#1931)
* chore(prom): rename prometheus(remote storage) to prom-store and promql(HTTP server) to prometheus

* chore: apply clippy suggestions

* chore: adjust format according to rustfmt
2023-07-12 14:47:09 +08:00
Weny Xu
2d30f4c373 fix: fix broken CI (#1933) 2023-07-11 14:48:41 +08:00
Eugene Tolbakov
fc850c9988 feat(config-endpoint): add initial implementation (#1896)
* feat(config-endpoint): add initial implementation

* feat: add initial handler implementation

* fix: apply clippy suggestions, use axum response instead of string

* feat: address CR suggestions

* fix: minor adjustments in formatting

* fix: add a test

* feat: add to_toml_string method to options

* fix: adjust the assertion for the integration test

* fix: adjust expected indents

* fix: adjust assertion for the integration test

* fix: improve according to clippy
2023-07-11 11:08:32 +08:00
Lei, HUANG
f287d3115b chore: replace result assertions (#1840)
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g

* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
2023-06-27 19:14:48 +08:00
Niwaka
a103614fd2 feat: support /api/v1/series for Prometheus (#1620)
* feat: support /api/v1/series for Prometheus

* chore: error handling

* feat: update tests
2023-06-06 10:29:16 +08:00
Chuanle Chen
77497ca46a feat: support /api/v1/label/<label_name>/values from Prometheus (#1604)
* feat: support `/api/v1/label/<label_name>/values` from Prometheus

* chore: apply CR

* chore: apply CR
2023-05-22 07:24:12 +00:00
Huaijin
715e1a321f feat: implement /api/v1/labels for prometheus (#1580)
* feat: implement /api/v1/labels for prometheus

* fix: only gather match[]

* chore: fix typo

* chore: fix typo

* chore: change style

* fix: suggestion

* fix: suggestion

* chore: typo

* fix: fmt

* fix: add more test
2023-05-17 03:56:22 +00:00
Ning Sun
497b1f9dc9 feat: metrics for storage engine (#1574)
* feat: add storage engine region count gauge

* test: remove catalog metrics because we can't get a correct number

* feat: add metrics for log store write and compaction

* fix: address review issues
2023-05-15 15:22:00 +08:00
Ning Sun
b0ab641602 feat: add catalog/schema/table count as catalog metrics (#1499)
* feat: add catalog/schema/table count as catalog metrics

* test: add integration tests for catalog metrics
2023-05-05 05:54:12 +00:00
Yingwen
eb50cee601 feat: Switch to the procedure framework (#1448)
* feat: Remove create_mock_sql_handler()

create_to_request() and alter_to_request() don't need `&self`, so
we don't need to mock the sql handler to test them

* feat: Enable procedure manager by default

* docs: Update config example

* test: Enable procedure framework in all tests

* refactor(datanode): rename methods using procedure

* test(catalog): Fix temp dir drops before test finishes

* tests: Enable procedure framework in sqlness

* test: Fix sqlness standalone rename test

* fix: Drop procedure allows table not in engine

* test: Change rename table test

* fix: add options to table meta when creating table by procedure

* test: adjust error message in schema test case

* test: Fix test_sql_api error message
2023-04-25 12:04:02 +08:00
Ruihang Xia
b6647af2e3 test: add integration case to check dashboard path (#1422)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-20 11:17:01 +08:00
Ruihang Xia
e8bb00f0be feat: impl instant query interface (#1410)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-04-18 23:25:14 +08:00
LFC
48c2841e4d feat: execute python script in distributed mode (#1264)
* feat: execute python script in distributed mode

* fix: rebase develop
2023-04-02 20:36:48 +08:00