Commit Graph

529 Commits

Author SHA1 Message Date
Jeremyhi
df0fff2f2c feat(servers): make http timeout and body limit optional (#4217)
* feat(servers): make http timeout and body limit optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: make config-docs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-26 06:14:14 +00:00
dennis zhuang
a779cb36ec fix: wrong frontend registration address (#4199)
* fix: frontend registration address is wrong, #4186

* fix: license header

* chore: adds hostname to frontend grpc

* fix: forgot run make config-docs

* chore: warn when using bind_addr

* fix: flow node heartbeat carrying address
2024-06-26 06:13:07 +00:00
ZonaHe
dea33a7aaf feat: update dashboard to v0.5.2 (#4185)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2024-06-21 14:32:46 +00:00
Yohan Wal
b739c9fd10 feat: PREPARE and EXECUTE statement from mysql client (#4125)
* feat: prepare stmt in mysql client

* feat: execute stmt in mysql client

* fix: handle parameters properly

* refactor: use existing funcs to convert expr to scalar value

* refactor: use uuid strings as stmt_key for queries from COM_PREPARE packet

* refactor: take prepare and execute parser as submodule

* test: add unit test for converting expr to scalar value

* feat: deallocate stmt in mysql client

* chore: comments and duplicates

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-06-21 02:02:57 +00:00
Ruihang Xia
21c89f3247 perf: optimize RecordBatch to HttpOutput conversion (#4178)
* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 70ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add profiler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 50ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 160ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix windows build

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-20 12:33:58 +00:00
localhost
48a0f39b19 chore: enhance add pipeline http api return data (#4167)
* chore: enhance add pipeline http api return data

* chore: replaceing hard code header value
2024-06-20 02:19:31 +00:00
ZonaHe
8abebad458 feat: update dashboard to v0.5.1 (#4171)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2024-06-20 01:46:54 +00:00
taobo
22d12683b4 refactor!: unify FrontendOptions and DatanodeOptions by using GrpcOptions (#4088)
* refactor: move GrpcOptions to servers/grpc

* fix: optimize code

* fix: docs

* refactor: move DatanodeOptions.rpc_hostname to grpc.hostname

* fix: merge main

* refactor code impl

test: add test_depreacted_cli_options unit test

* Update src/servers/src/grpc.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-06-18 22:45:38 +00:00
localhost
01e3a24cf7 feat: log ingestion support (#4014)
* chore: add log http ingester scaffold

* chore: add some example code

* chore: add log inserter

* chore: add log handler file

* chore: add pipeline lib

* chore: import log handler

* chore: add pipelime http handler

* chore: add pipeline private table

* chore: add pipeline API

* chore: improve error handling

* chore: merge main

* chore: add multi content type support for log handler

* refactor: remove servers dep on pipeline

* refactor: move define_into_tonic_status to common-error

* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4

* chore: fix typo

* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c

* chore: fix typo and license header

* refactor: move http event handler to a separate file

* chore: add test for pipeline

* chore: fmt

* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93

* refactor: move `pipeline_operator` to `pipeline` crate

* chore: minor update

* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4

* chore: add log

* chore: add log

* chore: remove open hook

* chore: minor update

* chore: fix fmt

* chore: minor update

* chore: rename desc for pipeline table

* refactor: remove updated_at in pipelines

* chore: add more content type support for log inserter api

* chore: introduce pipeline crate

* chore: update upload pipeline api

* chore: fix by pr commit

* chore: add some doc for pub fn/struct

* chore: some minro fix

* chore: add pipeline version support

* chore: impl log pipeline version

* chore: fix format issue

* fix: make the LogicalPlan of a query pipeline sorted in desc order

* chore: remove some debug log

* chore: replacing hashmap cache with moak

* chore: fix by pr commit

* chore: fix toml format issue

* chore: update Cargo.lock

* chore: fix by pr commit

* chore: fix some issue by pr commit

* chore: add more doc for pipeline version

---------

Co-authored-by: shuiyisong <xixing.sys@gmail.com>
2024-06-14 17:03:30 +00:00
Jeremyhi
4719569e4f feat: support gRPC cancellation (#4092)
* feat: support cancellation

* chore: add unit test for cancellation

* chore: minor refactor

* feat: we do not need to spawn in distributed mode

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-06-06 08:27:25 +00:00
shuiyisong
d1838fb28d refactor: move define_into_tonic_status to common-error (#4095)
* chore: finish cherry-pick

* chore: remove unused code
2024-06-04 03:29:15 +00:00
Ruihang Xia
c2218f8be8 build(deps): bump datafusion 20240528 (#4061)
* build(deps): bump datafusion 20240528

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* another update

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update expected sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix first/last value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reformat comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remaining errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert toml format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 feature

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dead code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* format file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
2024-06-01 14:03:00 +00:00
Ruihang Xia
45fee948e9 fix: display error in correct format (#4082)
* fix: display error in correct format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add address to RegionServer error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-31 09:25:14 +00:00
dennis zhuang
aafb468547 fix: set local or session time_zone not work (#4064)
* fix: set local or session time_zone not work

* chore: supports PostgreSQL-specific setting time zone
2024-05-29 00:06:13 +00:00
tison
9dd6e033a7 refactor: move Database to client crate behind testing feature (#4059)
* refactor: move Database to client crate behind testing feature

Signed-off-by: tison <wander4096@gmail.com>

* partial move

Signed-off-by: tison <wander4096@gmail.com>

* catch up more

Signed-off-by: tison <wander4096@gmail.com>

* fix imports

Signed-off-by: tison <wander4096@gmail.com>

* finish

Signed-off-by: tison <wander4096@gmail.com>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2024-05-28 03:21:43 +00:00
tison
f9db5ff0d6 build(deps): upgrade opendal to 0.46 (#4037)
* build(deps): upgrade opendal to 0.46

Signed-off-by: tison <wander4096@gmail.com>

* migrate writes

Signed-off-by: tison <wander4096@gmail.com>

* migrate reads

Signed-off-by: tison <wander4096@gmail.com>

* fixup object safety

Signed-off-by: tison <wander4096@gmail.com>

* fixup names

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* a few Buffer to Vec

Signed-off-by: tison <wander4096@gmail.com>

* Make greptime buildable with opendal 0.46 (#5)

Signed-off-by: Xuanwo <github@xuanwo.io>

* fixup toml check

Signed-off-by: tison <wander4096@gmail.com>

* test_orc_opener

Signed-off-by: tison <wander4096@gmail.com>

* Fix lru cache (#6)

Signed-off-by: Xuanwo <github@xuanwo.io>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* improve comments

Signed-off-by: tison <wander4096@gmail.com>

* address comments

Signed-off-by: tison <wander4096@gmail.com>

* reduce buf copy

Signed-off-by: tison <wander4096@gmail.com>

* upgrade to reqwest 0.12

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Co-authored-by: Xuanwo <github@xuanwo.io>
2024-05-27 09:12:23 +00:00
tison
c78043d526 build(deps): merge tower deps to workspace (#4036)
Signed-off-by: tison <wander4096@gmail.com>
2024-05-27 07:15:30 +00:00
LFC
297105266b feat: enable tcp keepalive for http server (#4019)
* feat: enable tcp keepalive for http server

* chore: for enterprise's update

* resolve PR comments
2024-05-27 04:07:36 +00:00
tison
3790020d78 build(deps): upgrade promql-parser to 0.4 (#4047)
* build(deps): upgrade promql-parser to 0.4

Signed-off-by: tison <wander4096@gmail.com>

* lock

Signed-off-by: tison <wander4096@gmail.com>

* catch up upgrades

Signed-off-by: tison <wander4096@gmail.com>

* concise method

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2024-05-27 01:51:59 +00:00
taobo
a3a2c8d063 feat: Add TLS support for gRPC service (#3957)
* feat: Add tls support for grpc service

* feat: add integration test

* fix: integration test

* fix: revert by suggestion

* fix: typos

* fix: optimize code

* fix: optimize code

* docs: update configs
2024-05-23 19:00:16 +00:00
Ning Sun
b90b7adf6f feat: add fallback logic for vmagent sending wrong content type (#4009)
* feat: add fallback logic for vmagent sending wrong content type

* fix: resolve lint issues

* Update src/servers/src/http/prom_store.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-05-23 02:40:17 +00:00
Jeremyhi
418090b464 chore: log error for detail (#4011)
* chore: log error for detail

* chore: by cr
2024-05-22 12:17:20 +00:00
shuiyisong
19543f9819 feat: support compression on gRPC server (#3961)
* feat: enable gzip in grpc server side

* feat: add enable_gzip_compression config

* test: add grpc compression test

* feat: support user configured compression on grpc server

* chore: update doc

* chore: add tests

* fix: make config-docs

* chore: fix cr issue

* chore: add test

* refactor: remove config on server side, auto enable all compression support

* chore: minor update

* chore: remove unused code

* refactor: enable zstd compression internally by default

* chore: minor fix
2024-05-20 11:28:00 +00:00
Eugene Tolbakov
6a9a92931d chore: change binary array type from LargeBinaryArray to BinaryArray (#3924)
* chore: change binary array type from LargeBinaryArray to BinaryArray

* fix: adjust try_into_vector logic

* fix: apply CR suggestions, add tests

* chore: fix failing test

* chore: fix integration test

* chore: adjust the assertions according to changed implementation

* chore: add a test with LargeBinary type

* chore: apply CR suggestions

* chore: simplify tests
2024-05-18 08:04:41 +00:00
Ning Sun
fe34ebf770 test: give windows file watcher more time (#3953)
* test: give windows file watcher more time

* refactor: use constants for timeout
2024-05-16 01:58:45 +00:00
Yingwen
529e344450 ci: Use lld linker in windows tests (#3946)
* ci: disable other test

* ci: timeout 30

* ci: try to use lld

* ci: change linker

* test: wait for file change in test multiple times

* ci: enable other tests

* chore: revert sleep in loop
2024-05-15 12:34:10 +00:00
maco
494ce65729 feat: limiting the size of query results to Dashboard (#3901)
* feat: limiting the size of query results to Dashboard

* optimize code

* fix by cr

* fix integration tests error

* remove RequestSource::parse

* refactor: sql query params

* fix: unit test

---------

Co-authored-by: tison <wander4096@gmail.com>
2024-05-14 01:57:30 +00:00
Ruihang Xia
590aedd466 fix: sort unstable HTTP result in label values query (#3920)
* fix: sort unstable HTTP result in label values query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: Update src/servers/src/http/prometheus.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-05-11 11:33:01 +00:00
Yingwen
d0820bb26d refactor: Remove PhysicalPlan trait and use ExecutionPlan directly (#3894)
* refactor: remove PhysicalPlan

* refactor: remove physical_plan mod

* refactor: import

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-11 07:38:03 +00:00
Ruihang Xia
115c74791d build(deps): bump snafu to 0.8 (#3911)
* change Cargo.toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias in script engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-10 13:36:25 +00:00
Ruihang Xia
08945f128b fix: sort unstable HTTP result on test profile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-10 17:01:42 +08:00
Weny Xu
5545a8b023 feat: implement drop flow procedure (#3877)
* feat: implement `destroy_flow_metadata` method

* chore: bump proto to 65c1364

* feat: implement the drop flow procedure

* feat: add `MockFlownodeManager`

* tests: add tests for create flow & drop flow procedure

* chore: apply suggestions from CR

* chore: use `ClusterId`
2024-05-09 08:23:19 +00:00
Jeremyhi
f995f6099f chore: avoid double future (#3890) 2024-05-09 07:11:22 +00:00
Ruihang Xia
a6a702de4e feat: support querying field column names in Prometheus HTTP API (#3880)
* feat: support querying field column names in Prometheus HTTP API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use tables stream API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-08 12:18:20 +00:00
Ruihang Xia
71c1c7ca24 fix: return metric name instead of query in Prometheus /series API (#3864)
* fix: return metric name instead of query in Prometheus /series API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* omit non-tag columns from result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-07 09:07:16 +00:00
Kelvin Wu
154f561da1 refactor: passing QueryContext to RegionServer (#3829)
* refactor: passing QueryContext to RegionServer

* refactor: change the return type of build() in QueryContextBuilder

* fix: update greptime-proto reference

* chore: apply suggestion

* chore: revert the last commit

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-05-07 15:10:59 +08:00
shuiyisong
1b93a026c2 ci: align clippy checks and fixup (#3868)
* fix: make clippy

* fix: use make in CI
2024-05-07 03:00:38 +00:00
Ruihang Xia
530353785c refactor: remove re-export from logging (#3865)
* refactor: remove re-export from logging

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge problem

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* run formatter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-06 13:26:01 +00:00
shuiyisong
6e9e8fad26 refactor!: remove opentsdb tcp server (#3828)
* refactor: remove opentsdb tcp server

* refactor: remove config and add test

* refactor: update docs and remove unused code
2024-05-06 06:42:05 +00:00
dennis zhuang
65d47bab56 feat: adds information_schema cluster_info table (#3832)
* feat: adds server running mode to KvBackendCatalogManager

* feat: adds MetaClient to KvBackendCatalogManager

* feat: impl information_schema.cluster_info table

* fix: forgot files

* test: update information_schema result

* feat: adds start_time and uptime to cluster_info

* chore: tweak cargo and comment

* feat: rename greptime_region_peers to region_peers

* fix: cluster_info result

* chore: simplify sqlness commands

* chore: set peer_id to -1 for frontends

* fix: move cluster_info to greptime catalog

* chore: use official proto

* feat: adds active_time

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: STANDALONE for runtime_metrics

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
Co-authored-by: tison <wander4096@gmail.com>
2024-05-02 02:49:46 +00:00
dimbtp
3b89b9ddd8 refactor: move database client to test (#3820)
* remove `Database` in `cli export`

* move `Database` to tests-integration

* fix clippy

* move `DatabaseClient` along with `Database`

* `cli export` now use http api

* add TODO for `tests-integration` dependencies

* cleanup code

* 'cli export' test use http api

* remove unsed dependencies in cmd crate

* apply review comments and clean code

* remove unused methods

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-05-01 05:55:13 +00:00
tison
d11b1fa389 chore: correct RepeatedField origin (#3838)
* chore: correct RepeatedField origin

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2024-04-30 03:13:54 +00:00
tison
c387687262 refactor: use secrecy SerectString to hold secrets option (#3804)
* build: centralize secrecy dependency

Signed-off-by: tison <wander4096@gmail.com>

* add secrecy to sql crate

Signed-off-by: tison <wander4096@gmail.com>

* try impl

Signed-off-by: tison <wander4096@gmail.com>

* update test

Signed-off-by: tison <wander4096@gmail.com>

* make linters happy

Signed-off-by: tison <wander4096@gmail.com>

* bundle secrecy

Signed-off-by: tison <wander4096@gmail.com>

* bundle secrecy

Signed-off-by: tison <wander4096@gmail.com>

* replace secrecy

Signed-off-by: tison <wander4096@gmail.com>

* tidy clones

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* updated

Signed-off-by: tison <wander4096@gmail.com>

* Apply suggestions from code review

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* use BTreeMap

Signed-off-by: tison <wander4096@gmail.com>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
2024-04-29 02:18:18 +00:00
Ruihang Xia
c0b909330a fix: wrong handler implementation of prometheus remote write (#3826)
* fix: wrong handler implementation of prometheus remote write

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-28 10:52:32 +00:00
tison
d8ea7c5585 chore: respect axum test client's origin (#3805)
Signed-off-by: tison <wander4096@gmail.com>
2024-04-26 07:19:46 +00:00
tison
4eadd9f8a8 refactor: log statement on error by Display (#3803)
* refactor: log statement on error by Display

Signed-off-by: tison <wander4096@gmail.com>

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-04-26 04:05:13 +00:00
Jeremyhi
9524ec83bc fix!: use the right precision (#3794)
* fix: use write precision

* chore: rename error

* chore: add doc

* chore: by comment

* chore: right test result

* chore: typo

* chore: add doc
2024-04-25 03:20:10 +00:00
Ruihang Xia
1272bc9afc fix: post process result on query full column name of prom labels API (#3793)
* fix: post process result on query full column name of prom labels API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* only preserve tag column

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-24 09:38:03 +00:00
Ruihang Xia
f764fd5847 fix: consider both db param and extended db header in Prometheus HTTP API (#3776)
* fix: consider both db param and extended db header in Prometheus HTTP API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove debug code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-23 07:13:05 +00:00
ZonaHe
eeb4e26c71 feat: update dashboard to v0.5.0 (#3768)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-22 11:41:00 +00:00