rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-14 01:02:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,076 Commits 105 Branches 227 Tags
76dc906574953f3bc01ee700bb12edee1cd28e7d
Commit Graph

558 Commits

Author SHA1 Message Date
localhost
202c730363 perf: Optimizing pipeline performance (#4390) 
* chore: improve pipeline performance

* chore: use arc to improve time type

* chore: improve pipeline coerce

* chore: add vec refactor

* chore: add vec pp

* chore: improve pipeline

* inprocess

* chore: set log ingester use new pipeline

* chore: fix some error by pr comment

* chore: fix typo

* chore: use enum_dispatch to simplify code

* chore: some minor fix

* chore: format code

* chore: update by pr comment

* chore: fix typo

* chore: make clippy happy

* chore: fix by pr comment

* chore: remove epoch and date process add new timestamp process

* chore: add more test for pipeline

* chore: restore epoch and date processor

* chore: compatibility issue

* chore: fix by pr comment

* chore: move the evaluation out of the loop

* chore: fix by pr comment

* chore: fix dissect output key filter

* chore: fix transform output greptime value has order error

* chore: keep pipeline transform output order

* chore: revert tests

* chore: simplify pipeline prepare implementation

* chore: add test for timestamp pipelin processor

* chore: make clippy happy

* chore: replace is_some check to match

---------

Co-authored-by: shuiyisong <xixing.sys@gmail.com>
 2024-08-13 11:32:04 +00:00
shuiyisong
5aa4c70057 chore: update validator signature (#4548) 2024-08-13 08:06:12 +00:00
LFC
7dd20b0348 chore: make mysql server version changable (#4531) 2024-08-09 03:43:43 +00:00
ZonaHe
efe5eeef14 feat: update dashboard to v0.5.4 (#4483) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2024-08-01 12:19:38 +00:00
Jeremyhi
62a0defd63 feat: improve extract hints (#4479) 2024-08-01 07:06:13 +00:00
Jeremyhi
291d9d55a4 feat: hint options for gRPC insert (#4454) 
* feat: hint options for gRPC isnert

* chore: unit test for extract_hints

* feat: add integration test for grpc hint

* test: add integration test for hints
 2024-08-01 02:59:38 +00:00
shuiyisong
c66d3090b6 fix: prometheus api only returns 200 (#4471) 
fix: prometheus api returns http status other than 200
 2024-07-31 07:42:50 +00:00
Ning Sun
b741a7181b feat: track channels with query context and w/rcu (#4448) 
* feat: add source channel to meter recorders

* feat: provide channel for query context

* fix: testing and extension get for query context

* chore: revert cargo toml structure changes

* fix: querycontext modification for prometheus and pipeline

* chore: switch git dependency to main branches

* chore: remove TODO

* refactor: rename other to unknown

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
 2024-07-31 07:30:50 +00:00
Ran Miller
c24de8b908 refactor(servers): improve postgres error message (#4463) 
* refactor(servers): improve postgres error message

* refactor(servers): remove numerical representation of ErrorSeverity
 2024-07-31 06:06:15 +00:00
Ruihang Xia
7daf24c47f feat: remove dedicated runtime for grpc, mysql and pg protocols (#4436) 
* feat: remove dedicated runtime for grpc, mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove other runtimes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* spawn compact task into compact_runtime

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refine naming

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/servers/tests/mysql/mysql_server_test.rs

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* turnoff fuzz test matrix fail fast option

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update rt config for ci tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-30 06:17:58 +00:00
shuiyisong
567f5105bf fix: missing pre_write check on prometheus remote write (#4460) 
fix: missing pre_write check on prometheus remote write
 2024-07-30 04:55:19 +00:00
shuiyisong
53fc14a50b fix: use status code to http status mapping in error IntoResponse (#4455) 2024-07-29 16:37:04 +00:00
Ruihang Xia
1895a5478b feat: track prometheus HTTP API's query latency (#4458) 
* feat: track prometheus HTTP API's query latency

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update grafana config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: Update src/servers/src/metrics.rs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-07-29 15:00:09 +00:00
Ning Sun
b298b35b3b feat: show root cause and db name on the error line (#4442) 
* feat: show root cause on the error line

* feat: show root error for grpc

* feat: add error information for http error

* feat: add db information on error mysql/postgres logs
 2024-07-29 03:59:42 +00:00
discord9
021ec7b6ac feat(flow): flush_flow function (#4416) 
* refactor: df err variant

* WIP

* chore: update proto version

* chore: revert mistaken rust-toolchain

* feat(WIP): added FlowService to QueryEngine

* refactor: move flow service to operator

* refactor: flush use flow name not id

* refactor: use full path in macro

* feat: flush flow

* feat: impl flush flow

* chore: remove unused

* chore: meaninful response

* chore: remove unused

* chore: clippy

* fix: flush_flow with proper blocking

* test: sqlness tests added back for flow

* test: better predicate for flush_flow

* refactor: rwlock

* fix: flush lock

* fix: flush lock write then drop

* test: add a new flow sqlness test

* fix: sqlness testcase

* chore: style

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-26 23:04:13 +00:00
shuiyisong
547730a467 chore: add metrics for log ingestion (#4411) 
* chore: add metrics for log ingestion

* chore: record result as well
 2024-07-23 08:05:11 +00:00
dennis zhuang
be3ea0fae7 feat: improve datafusion external error and mysql error (#4362) 
* feat: improve datafusion external error and mysql error

* chore: address CR comments and fix tests

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
 2024-07-16 07:01:09 +00:00
dennis zhuang
64cad4e891 feat: tweak error and status codes (#4359) 
* feat: tweak status codes

* fix: typo

* fix: by cr comments
 2024-07-15 07:50:16 +00:00
shuiyisong
67dfdd6c61 feat: support text/plain format for log ingestion (#4300) 
* feat: support text/plain format of log input

* refactor: pipeline query and delete using dataframe api

* chore: minor refactor

* refactor: skip jsonify when processing plan/text

* refactor: support array(string) as pipeline engine input
 2024-07-12 09:17:15 +00:00
tison
285ffc5850 fix: build info should use build time env var (#4343) 
* fix: build info should use build time env var

Signed-off-by: tison <wander4096@gmail.com>

* catch up

Signed-off-by: tison <wander4096@gmail.com>

* fixup lifetime

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fix more

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-07-12 03:08:30 +00:00
Ning Sun
50e4539667 fix: permission denied is 403 (#4350) 2024-07-11 17:16:17 +00:00
Ruihang Xia
185953e586 fix: support unary operator in default value, partition rule and prepare statement (#4301) 
* handle unary operator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add prepare test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test and context

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rebase error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-09 08:59:06 +00:00
Zhenchi
00308218b3 feat(fulltext_index): allow enable full-text index in SQL and gRPC way (#4310) 
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: test_fulltext_intm_path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: explicitly build column options

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: fix error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-08 20:18:48 +00:00
localhost
c0e9b3dbe2 chore: disable TraceLayer on_failure log (#4315) 2024-07-08 10:53:35 +00:00
shuiyisong
849e0b9249 feat: delete pipeline (#4156) 
* feat: add delete for pipeline

* chore: remove unused code

* refactor: delete pipeline

* chore: add pipeline management api metrics

* chore: minor cr issues

* chore: add unit test

* chore: fix cr issue

* fix: test

* chore: add `GreptimedbManageResponse`

* fix: typo

* fix: typo
 2024-07-05 06:23:49 +00:00
Ning Sun
8399dcada3 refactor: use rwlock for modifiable session data (#4232) 
* chore: update sqlness results

* refactor: use rwlock for modifiable data in session and querycontext

* chore: format toml

* refactor: use mutable_inner structure for mutable fields

* refactor: remove arc wrapper
 2024-07-04 12:53:25 +00:00
ZonaHe
8e306f3f51 feat: update dashboard to v0.5.3 (#4262) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2024-07-03 14:45:55 +00:00
Ning Sun
11cf9c827e feat: dbeaver mysql compatibility, use statement and information_schema.tables (#4218) 
* feat: add more placeholder field in information_schema.tables

* feat: make schema modifiable for use statement

* chore: add todo items

* fix: resolve lint issues after data type changes

* chore: update sqlness results

* refactor: patch for select database is no longer needed

* test: align tests and data types

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* fix: use canonicalize_identifier for database name

* feat: add all columns for information_schema.tables

* test: remove vairables from sqlness results

* feat: add to_string impl for table options

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-03 01:31:13 +00:00
LFC
6276e006b9 refactor: add interceptor after Influxdb lines are converted to grpc row insert (#4225) 
* fix: make Influxdb lines able to be inserted into last created tables

* Update src/servers/src/influxdb.rs

* add an option to control the time index alignment behavior

* fix ci

* refactor: use interceptor to handle timestamp align

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: tison <wander4096@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-01 22:28:00 +00:00
Jeremyhi
df0fff2f2c feat(servers): make http timeout and body limit optional (#4217) 
* feat(servers): make http timeout and body limit optional

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: make config-docs

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-26 06:14:14 +00:00
dennis zhuang
a779cb36ec fix: wrong frontend registration address (#4199) 
* fix: frontend registration address is wrong, #4186

* fix: license header

* chore: adds hostname to frontend grpc

* fix: forgot run make config-docs

* chore: warn when using bind_addr

* fix: flow node heartbeat carrying address
 2024-06-26 06:13:07 +00:00
ZonaHe
dea33a7aaf feat: update dashboard to v0.5.2 (#4185) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2024-06-21 14:32:46 +00:00
Yohan Wal
b739c9fd10 feat: PREPARE and EXECUTE statement from mysql client (#4125) 
* feat: prepare stmt in mysql client

* feat: execute stmt in mysql client

* fix: handle parameters properly

* refactor: use existing funcs to convert expr to scalar value

* refactor: use uuid strings as stmt_key for queries from COM_PREPARE packet

* refactor: take prepare and execute parser as submodule

* test: add unit test for converting expr to scalar value

* feat: deallocate stmt in mysql client

* chore: comments and duplicates

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-06-21 02:02:57 +00:00
Ruihang Xia
21c89f3247 perf: optimize RecordBatch to HttpOutput conversion (#4178) 
* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 70ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add profiler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 50ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 160ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix windows build

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-20 12:33:58 +00:00
localhost
48a0f39b19 chore: enhance add pipeline http api return data (#4167) 
* chore: enhance add pipeline http api return data

* chore: replaceing hard code header value
 2024-06-20 02:19:31 +00:00
ZonaHe
8abebad458 feat: update dashboard to v0.5.1 (#4171) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2024-06-20 01:46:54 +00:00
taobo
22d12683b4 refactor!: unify FrontendOptions and DatanodeOptions by using GrpcOptions (#4088) 
* refactor: move GrpcOptions to servers/grpc

* fix: optimize code

* fix: docs

* refactor: move DatanodeOptions.rpc_hostname to grpc.hostname

* fix: merge main

* refactor code impl

test: add test_depreacted_cli_options unit test

* Update src/servers/src/grpc.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-18 22:45:38 +00:00
localhost
01e3a24cf7 feat: log ingestion support (#4014) 
* chore: add log http ingester scaffold

* chore: add some example code

* chore: add log inserter

* chore: add log handler file

* chore: add pipeline lib

* chore: import log handler

* chore: add pipelime http handler

* chore: add pipeline private table

* chore: add pipeline API

* chore: improve error handling

* chore: merge main

* chore: add multi content type support for log handler

* refactor: remove servers dep on pipeline

* refactor: move define_into_tonic_status to common-error

* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4

* chore: fix typo

* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c

* chore: fix typo and license header

* refactor: move http event handler to a separate file

* chore: add test for pipeline

* chore: fmt

* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93

* refactor: move `pipeline_operator` to `pipeline` crate

* chore: minor update

* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4

* chore: add log

* chore: add log

* chore: remove open hook

* chore: minor update

* chore: fix fmt

* chore: minor update

* chore: rename desc for pipeline table

* refactor: remove updated_at in pipelines

* chore: add more content type support for log inserter api

* chore: introduce pipeline crate

* chore: update upload pipeline api

* chore: fix by pr commit

* chore: add some doc for pub fn/struct

* chore: some minro fix

* chore: add pipeline version support

* chore: impl log pipeline version

* chore: fix format issue

* fix: make the LogicalPlan of a query pipeline sorted in desc order

* chore: remove some debug log

* chore: replacing hashmap cache with moak

* chore: fix by pr commit

* chore: fix toml format issue

* chore: update Cargo.lock

* chore: fix by pr commit

* chore: fix some issue by pr commit

* chore: add more doc for pipeline version

---------

Co-authored-by: shuiyisong <xixing.sys@gmail.com>
 2024-06-14 17:03:30 +00:00
Jeremyhi
4719569e4f feat: support gRPC cancellation (#4092) 
* feat: support cancellation

* chore: add unit test for cancellation

* chore: minor refactor

* feat: we do not need to spawn in distributed mode

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-06 08:27:25 +00:00
shuiyisong
d1838fb28d refactor: move define_into_tonic_status to common-error (#4095) 
* chore: finish cherry-pick

* chore: remove unused code
 2024-06-04 03:29:15 +00:00
Ruihang Xia
c2218f8be8 build(deps): bump datafusion 20240528 (#4061) 
* build(deps): bump datafusion 20240528

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* another update

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update expected sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix first/last value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reformat comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remaining errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert toml format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 feature

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dead code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* format file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-06-01 14:03:00 +00:00
Ruihang Xia
45fee948e9 fix: display error in correct format (#4082) 
* fix: display error in correct format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add address to RegionServer error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-31 09:25:14 +00:00
dennis zhuang
aafb468547 fix: set local or session time_zone not work (#4064) 
* fix: set local or session time_zone not work

* chore: supports PostgreSQL-specific setting time zone
 2024-05-29 00:06:13 +00:00
tison
9dd6e033a7 refactor: move Database to client crate behind testing feature (#4059) 
* refactor: move Database to client crate behind testing feature

Signed-off-by: tison <wander4096@gmail.com>

* partial move

Signed-off-by: tison <wander4096@gmail.com>

* catch up more

Signed-off-by: tison <wander4096@gmail.com>

* fix imports

Signed-off-by: tison <wander4096@gmail.com>

* finish

Signed-off-by: tison <wander4096@gmail.com>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-05-28 03:21:43 +00:00
tison
f9db5ff0d6 build(deps): upgrade opendal to 0.46 (#4037) 
* build(deps): upgrade opendal to 0.46

Signed-off-by: tison <wander4096@gmail.com>

* migrate writes

Signed-off-by: tison <wander4096@gmail.com>

* migrate reads

Signed-off-by: tison <wander4096@gmail.com>

* fixup object safety

Signed-off-by: tison <wander4096@gmail.com>

* fixup names

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* a few Buffer to Vec

Signed-off-by: tison <wander4096@gmail.com>

* Make greptime buildable with opendal 0.46 (#5)

Signed-off-by: Xuanwo <github@xuanwo.io>

* fixup toml check

Signed-off-by: tison <wander4096@gmail.com>

* test_orc_opener

Signed-off-by: tison <wander4096@gmail.com>

* Fix lru cache (#6)

Signed-off-by: Xuanwo <github@xuanwo.io>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* improve comments

Signed-off-by: tison <wander4096@gmail.com>

* address comments

Signed-off-by: tison <wander4096@gmail.com>

* reduce buf copy

Signed-off-by: tison <wander4096@gmail.com>

* upgrade to reqwest 0.12

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2024-05-27 09:12:23 +00:00
tison
c78043d526 build(deps): merge tower deps to workspace (#4036) 
Signed-off-by: tison <wander4096@gmail.com>
 2024-05-27 07:15:30 +00:00
LFC
297105266b feat: enable tcp keepalive for http server (#4019) 
* feat: enable tcp keepalive for http server

* chore: for enterprise's update

* resolve PR comments
 2024-05-27 04:07:36 +00:00
tison
3790020d78 build(deps): upgrade promql-parser to 0.4 (#4047) 
* build(deps): upgrade promql-parser to 0.4

Signed-off-by: tison <wander4096@gmail.com>

* lock

Signed-off-by: tison <wander4096@gmail.com>

* catch up upgrades

Signed-off-by: tison <wander4096@gmail.com>

* concise method

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-05-27 01:51:59 +00:00
taobo
a3a2c8d063 feat: Add TLS support for gRPC service (#3957) 
* feat: Add tls support for grpc service

* feat: add integration test

* fix: integration test

* fix: revert by suggestion

* fix: typos

* fix: optimize code

* fix: optimize code

* docs: update configs
 2024-05-23 19:00:16 +00:00
Ning Sun
b90b7adf6f feat: add fallback logic for vmagent sending wrong content type (#4009) 
* feat: add fallback logic for vmagent sending wrong content type

* fix: resolve lint issues

* Update src/servers/src/http/prom_store.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-05-23 02:40:17 +00:00