Commit Graph

59 Commits

Author SHA1 Message Date
zyy17
03a2e6d0c1 feat: support elasticsearch _bulk API to ingest logs (#5261)
* feat: support elasticsearch '_bulk' API to ingest logs

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

* refactor: add metrics

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2025-01-08 14:54:53 +00:00
yihong
369b59c84a fix: drop unused dep using udeps to minial the size (#5301)
* fix: drop unused dep using udeps to minial the size

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: adress comments fix the problem

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-08 06:36:22 +00:00
Ruihang Xia
548e1988ab refactor: remove unused symbols (#5193)
chore: remove unused symbols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-12-18 11:24:43 +00:00
Ning Sun
3133f3fb4e feat: add cursor statements (#5094)
* feat: add sql parsers for cursor operations

* feat: cursor operator

* feat: implement RecordBatchStreamCursor

* feat: implement cursor storage and execution

* test: add tests

* chore: update docstring

* feat: add a temporary sql rewrite for cast in limit

this issue is described in #5097

* test: add more sql for cursor integration test

* feat: reject non-select query for cursor statement

* refactor: address review issues

* test: add empty result case

* feat: address review comments
2024-12-06 09:32:22 +00:00
shuiyisong
c199604ece feat: Loki remote write (#4941)
* chore: add debug loki remote write url

* chore: add decode snappy

* chore: format output

* feature: impl loki remote write

* fix: special labels deserialize

* chore: move result to folder

* chore: finish todo in loki write

* test: loki write

* chore: fix cr issue

* chore: fix cr issue

* chore: fix cr issue

* chore: update pre-commit config

* chore: fix cr issue

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-11-18 08:39:17 +00:00
Lanqing Yang
cdba7b442f feat: implement statement/execution timeout session variable (#4792)
* support set and show on statement/execution timeout session variables.

* implement statement timeout for mysql read, and postgres queries

* add mysql test with max execution time
2024-11-15 06:19:39 +00:00
Ning Sun
e3c0b5482f feat: returning warning instead of error on unsupported SET statement (#4761)
* feat: add capability to send warning to pgclient

* fix: refactor query context to carry query scope data

* feat: return a warning for unsupported postgres statement
2024-09-24 08:45:55 +00:00
JohnsonLee
a8477e4142 fix: table resolving logic related to pg_catalog (#4580)
* fix: table resolving logic related to pg_catalog

refer to
https://github.com/GreptimeTeam/greptimedb/issues/3560#issuecomment-2287794348
and #4543

* refactor: remove CatalogProtocol type

* fix: sqlness

* fix: forbid create database pg_catalog with mysql client

* refactor: use QueryContext as arguments rather than Channel

* refactor: pass None as default behaviour in information_schema

* test: fix test
2024-09-09 00:47:59 +00:00
Ruihang Xia
64ae32def0 feat: remove some redundent clone/conversion on constructing MergeScan stream (#4632)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-08-28 08:52:09 +00:00
Ning Sun
b741a7181b feat: track channels with query context and w/rcu (#4448)
* feat: add source channel to meter recorders

* feat: provide channel for query context

* fix: testing and extension get for query context

* chore: revert cargo toml structure changes

* fix: querycontext modification for prometheus and pipeline

* chore: switch git dependency to main branches

* chore: remove TODO

* refactor: rename other to unknown

---------

Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
2024-07-31 07:30:50 +00:00
Ning Sun
8399dcada3 refactor: use rwlock for modifiable session data (#4232)
* chore: update sqlness results

* refactor: use rwlock for modifiable data in session and querycontext

* chore: format toml

* refactor: use mutable_inner structure for mutable fields

* refactor: remove arc wrapper
2024-07-04 12:53:25 +00:00
Ning Sun
11cf9c827e feat: dbeaver mysql compatibility, use statement and information_schema.tables (#4218)
* feat: add more placeholder field in information_schema.tables

* feat: make schema modifiable for use statement

* chore: add todo items

* fix: resolve lint issues after data type changes

* chore: update sqlness results

* refactor: patch for select database is no longer needed

* test: align tests and data types

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* fix: use canonicalize_identifier for database name

* feat: add all columns for information_schema.tables

* test: remove vairables from sqlness results

* feat: add to_string impl for table options

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-07-03 01:31:13 +00:00
Lei, HUANG
090b59e8d6 feat: manual compaction (#3988)
* add compaction udf params

* wip: pass compaction options through grpc

* wip: pass compaction options all the way down to region server

* wip: window compaction task

* feat: trigger major compaction

* refactor: optimize compaction parameter parsing

* chore: rebase main

* chore: update proto

* chore: add some tests

* feat: validate catalog

* chore: fix typo and rebase main

* fix: some cr comments

* fix: file_time_bucket_span

* fix: avoid upper bound overflow

* chore: update proto
2024-05-22 09:42:21 +00:00
Ruihang Xia
115c74791d build(deps): bump snafu to 0.8 (#3911)
* change Cargo.toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias in script engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-05-10 13:36:25 +00:00
Weny Xu
0e05f85a9d feat: pass QueryContext to FlowRequestHeader (#3878)
* feat: pass `QueryContext` to `DdlTaskRequest`

* feat: pass `QueryContext` to `FlowRequestHeader`

* chore: fmt toml
2024-05-09 04:57:33 +00:00
Kelvin Wu
154f561da1 refactor: passing QueryContext to RegionServer (#3829)
* refactor: passing QueryContext to RegionServer

* refactor: change the return type of build() in QueryContextBuilder

* fix: update greptime-proto reference

* chore: apply suggestion

* chore: revert the last commit

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-05-07 15:10:59 +08:00
Ruihang Xia
f764fd5847 fix: consider both db param and extended db header in Prometheus HTTP API (#3776)
* fix: consider both db param and extended db header in Prometheus HTTP API

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove debug code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-04-23 07:13:05 +00:00
LFC
314f2704d4 build(deps): update datafusion to latest and arrow to 51.0 (#3661)
* chore: update datafusion

* update sqlness case of time.sql

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: adjust range query partition

* fix: hisogram incorrect result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: ignore filter pushdown temporarily

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update limit sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: histogram with wrong distribution

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update negative ordinal sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: bump df to cd7a00b

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* ignore test_range_filter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix promql exec panic

* fix "select count(*)" exec error

* re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed

* fix: range query schema error

* update sqlness results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* update datafusion, again

* fix pyo3 compile error, and update some sqlness results

* update decimal sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: promql literal

* fix udaf tests

* fix filter pushdown sqlness tests

* fix?: test_cast

* fix: rspy test fail due to datafusion `sin` signature change

* rebase main to see if there are any failed tests

* debug ci

* debug ci

* debug ci

* enforce input partition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* debug ci

* fix ci

* fix ci

* debug ci

* debug ci

* debug ci

* fix sqlness

* feat: do not return error while creating a filter

* chore: remove array from error

* chore: replace todo with unimplemented

* Update src/flow/clippy.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WUJingdi <taylor-lagrange@qq.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-04-18 12:07:18 +00:00
JohnsonLee
d6b2d1dfb8 feat: Support outputting various date styles for postgresql (#3602)
* test: add integration_test for datetime style

* feat: support various datestyle for postgres

* doc: rewrite the comment about merge_datestyle_value

* test: add more test to illustrate valid datestyle input
2024-04-01 07:31:36 +00:00
Ning Sun
500f9f10fc feat: allow cross-schema query in promql (#3545)
* feat: add __schema__ tag for promql parser

* feat: disable matcher op other than equals

* test: add more test to ensure context getting reset

* test: add integration test

* test: refactor tests

* refactor: remove duplicated test code

* refactor: update according to review comments

* test: add sqlness test for cross schema scenario

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-03-29 07:41:01 +00:00
dennis zhuang
e3b37ee2c9 fix: canonicalize catalog and schema names (#3600) 2024-03-28 06:40:15 +00:00
JohnsonLee
83643eb195 feat: Support printing postgresql's bytea data type in its "hex" and "escape" format (#3567)
* feat: support set variable statement of session

* feat: support printing postgresql's bytea data type in its "hex" and "escape" format in ugly way

* refactor: add 'SessionConfigValue' type and unify the name

* doc: add license header

* refactor: confine coupling with 'sql::ast::Value' in SessionConfigValue

* refactor: move all bytea wrapper into bytea.rs

* fix: remove unused import in context.rs and postgres.rs

* refactor: rename 'set_configuration_parameter' to 'set_session_config'

rename 'set_configuration_parameter' in statement_.rs to 'set_session_config'

* refactor: use mod to organize options via macro

* refactor: re-model the session config value with static type

* test: add integration test

* refactor: move the encode bytea by format type logic into encoder

refactor: use Arc<DashMap> instead of DashMap in QueryContext

refactor: use Arc<DashMap> instead of DashMap in QueryContext

    Avoid expensive clone

refactor: use unreachable!() instead of unimplemented!()

refactor: move the encode bytea by format type logic into encoder

test: add binary format integration test case

* test: add ut for byte related type

* doc: remove TODO of bytea_output

* refactor: simplify the implementation with simple struct instead of complex typing

* fix: typo of 'Available'

* fix compile

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-03-27 01:54:41 +00:00
dimbtp
dd18d8c97b build(deps): remove some unused dependencies (#3582)
* build(deps): remove some unused dependencies

* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
Kould
00cbbc97ae feat: support Create Table ... Like (#3372)
* feat: support `Create Table ... Like`

* fix: `check_permission` for `Create Table ... Like`

* style: renaming `name` -> `table_name` & `target` -> `source_name` and make `Create Table ... Like` testcase more complicated

* rebase

* avoid _ fn

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
2024-03-02 06:34:13 +00:00
dennis zhuang
4b36c285f1 feat: flush or compact table and region functions (#3363)
* feat: adds Requester to process table flush and compaction request

* feat: admin_fn macros for administration functions

* test: add query result

* feat: impl flush_region, flush_table, compact_region, and flush_region functions

* docs: add Arguments to admin_fn macro

* chore: apply suggestion

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: apply suggestion

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: group_requests_by_peer and adds log

* Update src/common/macro/src/admin_fn.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* feat: adds todo for spawan thread

* feat: rebase with main

---------

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-02-27 08:57:38 +00:00
SteveLauC
e9a2b0a9ee chore: use workspace-wide lints (#3352)
* chore: use workspace-wide lints

* respond to review
2024-02-22 01:01:10 +00:00
shuiyisong
4cbdf64d52 chore: start plugins during standalone startup & comply with current catalog while changing database (#3282)
* chore: start plugins in standalone

* chore: respect current catalog in use statement for mysql

* chore: reduce unnecessory convert to string

* chore: reduce duplicate code
2024-02-06 02:41:37 +00:00
Ruihang Xia
31787f4bfd feat!: switch prom remote write to metric engine (#3198)
* feat: switch prom remote write to metric engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* fix compile

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* read physical table name from url

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove physical table from header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add with_metric_engine option to config remote write behavior

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* check parameter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add specific config param

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* default with_metric_engine to true

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update UT

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2024-01-22 14:48:28 +00:00
dennis zhuang
5e89472b2e feat: adds parse options for SQL parser (#3193)
* feat: adds parse options for parser and timezone to scan request

* chore: remove timezone in ScanRequest

* feat: remove timezone in parse options and adds type checking to parititon columns

* fix: comment

* chore: apply suggestions

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: format

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-01-19 09:16:36 +00:00
dennis zhuang
204b9433b8 feat: adds date_format function (#3167)
* feat: adds date_format function

* fix: compile error

* chore: use system timezone for FunctionContext and EvalContext

* test: as_formatted_string

* test: sqlness test

* chore: rename function
2024-01-17 03:24:40 +00:00
WU Jingdi
816d94892c feat: support HTTP&gRPC&pg set timezone (#3125)
* feat: support HTTP&gRPC&pg set timezone

* chore: fix code advice

* chore: fix code advice
2024-01-15 06:29:31 +00:00
WU Jingdi
d1ee1ba56a feat: support set timezone in db (#2992)
* feat: support set timezone in db

* chore: fix  ci

* chore: fix code advice

* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
WU Jingdi
500e299e40 feat: Enable distributed tracing in greptimedb (#2755)
* feat: implement distributed tracing

* fix: change usage of span

* fix: use otlp as exporter

* chore: update dependence

* chore: add span info

* chore: add alias

* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
tison
0cd6dacb45 chore: multiple licenses fixes (#2714)
* chore: avoid license risk from akka

Signed-off-by: tison <wander4096@gmail.com>

* chore: LICENSE test is AS IS

Signed-off-by: tison <wander4096@gmail.com>

* ci: use hawkeye to pin in inceptionYear

Signed-off-by: tison <wander4096@gmail.com>

* add instruction for running license header apply

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2023-11-09 10:38:12 +00:00
Ruihang Xia
93f21b188d chore: unify workspace dependencies and remove some unused (#2713)
* log-store

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* base64

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* udeps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* snafu style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change the workspace deps' style

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2023-11-09 07:34:22 +00:00
shuiyisong
067917845f fix: carry dbname from frontend to datanode (#2520)
* chore: add dbname in region request header for tracking purpose

* chore: fix handle read

* chore: add write meter

* chore: add meter-core to dep

* chore: add converter between RegionRequestHeader and QueryContext & update proto version
2023-10-08 06:30:23 +00:00
Ning Sun
0ad3fb6040 fix: mysql timezone settings (#2534)
* fix: restore time zone settings for mysql

* test: add integration test for time zone

* test: fix unit test for check
2023-10-07 10:21:32 +00:00
shuiyisong
7f51141ed0 refactor: auth crate (#2148)
* chore: move user_info to auth crate

* chore: temp commit before resolving tests compile error

* chore: fix compile issue

* chore: minor fix

* chore: tmp save

* chore: change user_info to trait

* chore: minor change & use auth result user info in pg session setup

* chore: add as_any to user_info

* chore: rename user_info

* chore: remove ice file

* chore: add permission checker

* chore: add grpc permission check

* chore: add session spawn user_info to query_ctx

* chore: minor update

* chore: add permission checker to sql handler & temp save

* chore: add permission checker to prometheus handler

* chore: add permission checker to opentsdb handler

* chore: add permission checker to other handlers

* chore: add test

* chore: add user_info setting on http entrance

* chore: fix toml

* chore: remove box in permission req

* chore: cr issue

* chore: cr issue
2023-08-14 02:51:26 +00:00
gongzhengyang
ce0f909cac perf: change current schema and catalog to borrow, clone only necessary (#2116)
perf: change current schema and catalog to borrow, clone only when necessary

Co-authored-by: gongzhengyang <gongzhengyang@bolean.com.cn>
2023-08-08 12:48:24 +00:00
Ning Sun
c39de9072f refactor: use workspace dependencies for internal modules (#2119)
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
2023-08-08 11:02:34 +00:00
LFC
46fa3eb629 chore: upgrade rust toolchain to latest nightly (#2049)
* chore: upgrade rust toolchain to latest nightly

* rebase develop

* update rust toolchain in ci
2023-08-08 07:17:51 +00:00
shuiyisong
6f1094db0a fix: arc() usage in non-test code (#2091)
* chore: try fix arc issue

* chore: move `parse_catalog_and_schema_from_client_database_name` to catalog crate

* fix: arc issue

* fix: arc issue

* fix: arc issue

* fix: arc issue

* fix: minor change
2023-08-03 10:16:02 +00:00
shuiyisong
9c1f0234de refactor: query context (#2022)
* chore: refactor query_context

* chore: remove use statement

* chore: make query_context build return arc

* fix: sqlness test

* fix: cr issue

* fix: use unwrap or else
2023-07-25 06:11:34 +00:00
shuiyisong
632cb26430 feat: trace_id in query context (#2014)
* chore: unify once_cell version

* chore: update cargo lock

* chore: add gen_trace_id

* chore: add trace_id to query_ctx

* chore: add debug log

* Revert "chore: add debug log"

This reverts commit f52ab3bb300f1d73117cd6ebbd8e0162829b1aba.

* chore: add frontend node id option

* chore: add query ctx to query engine ctx

* chore: set trace_id to logical_plan api

* chore: add trace_id in grpc entrance

* chore: generate trace_id while creating query_ctx

* chore: fix typo

* chore: extract trace_id from grpc header

* chore: extract trace_id from grpc header

* chore: fix clippy

* chore: add `QueryContextBuilder`

* chore: change node_id in fe to string
2023-07-24 07:35:06 +00:00
Ning Sun
e0aecc9209 refactor: improve semantics of session and query context (#2009) 2023-07-21 03:50:32 +00:00
LFC
fcff66e039 chore: deny unused results (#1825)
* chore: deny unused results

* rebase
2023-06-27 15:33:53 +08:00
dennis zhuang
ab5dfd31ec feat: sql dialect for different protocols (#1631)
* feat: add SqlDialect to query context

* feat: use session in postgrel handlers

* chore: refactor sql dialect

* feat: use different dialects for different sql protocols

* feat: adds GreptimeDbDialect

* refactor: replace GenericDialect with GreptimeDbDialect

* feat: save user info to session

* fix: compile error

* fix: test
2023-05-30 09:52:35 +08:00
Ning Sun
067c5ee7ce feat: time_zone variable for mysql connections (#1607)
* feat: add timezone info to query context

* feat: parse mysql compatible time zone string

* feat: add method to timestamp for rendering timezone aware string

* feat: use timezone from session for time string rendering

* refactor: use querycontectref

* feat: implement session/timezone variable read/write

* style: resolve toml format

* test: update tests

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update src/session/src/context.rs

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* refactor: address review issues

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-05-22 18:30:23 +08:00
Ning Sun
b0ab641602 feat: add catalog/schema/table count as catalog metrics (#1499)
* feat: add catalog/schema/table count as catalog metrics

* test: add integration tests for catalog metrics
2023-05-05 05:54:12 +00:00
Ning Sun
7dbac89000 feat: add metrics for protocol interfaces (#1495)
* feat: add metrics for various interfaces

* feat: add db label for protocols

* feat: add postgres protocol metrics

* feat: add metrics for grpcs apis

* feat: add auth failure counter for mysql/pg

* fix: add db label to grpc prometheus interface

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: add error code for auth failure counter

* fix: use schema as dbname when catalog is default

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
2023-04-28 23:42:35 +08:00