Commit Graph

119 Commits

Author SHA1 Message Date
Lei, Huang
2d869e1e43 refactor: datanode starts frontend (#471)
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend

* wip: start frontend in datanode

* wip: migrate create database to frontend

* wip: impl alter table

* fix: CR comments
2022-11-12 21:07:18 +08:00
LFC
f375e18a76 feat: table route cache (#462)
* feat: table route cache

Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-11 18:54:56 +08:00
dennis zhuang
e7b4d24df5 feat: create database (#451)
* feat: parsing create database statement

* feat: impl create database in datanode

* feat: supports insert into catalog.schema.table

* fix: conflicts with develop branch

* test: create database then insert and query

* fix: grpc schema provider

* feat: use CatalogManager::register_schema instead of CatalogProvide::register_schema

* refactor: revert InsertExpr catalog_name and schema_name

* fix: revert database.proto

* fix: revert client cargo

* feat: accepts schema.table as table name in sql

Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
2022-11-11 14:15:38 +08:00
Lei, Huang
952e646e1d refactor: move insert-to-create logic to a separate crate (#447) 2022-11-10 17:16:40 +08:00
Jiachun Feng
23f0320ffb feat: route for insert&select (#425)
* feat: route for insert/select

* chore: remove redundant tests

* chore: add fouce quit loop count limit to sequence

* chore: by code review

* chore: use ref with TableRouteKey

* chore: minor refactor
2022-11-10 16:13:15 +08:00
Lei, Huang
6288fdb6bc feat: frontend catalog (#437)
* feat: add frontend catalog
2022-11-10 11:52:57 +08:00
LFC
37dc85a29e feat: add gRPC reflection service (#431)
Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-09 19:13:01 +08:00
fys
d08f8b87a6 feat: convert different protocol to InsertRequest (#426)
* add line_writer and convert insert_stmt to InsertRequest

* support convert influxdb line protocol to InsertRequest

* support convert opentsdb to InsertRequest

* cr
2022-11-09 16:18:54 +08:00
Lei, Huang
2c9bcbe885 refactor: catalog crate (#415)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: databode lease

* feat: remote catalog (#356)

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: datanode heartbeat (#355)

* feat: add heartbeat task to instance

* feat: add node_id datanode opts

* fix: use real node id in heartbeat and meta client

* feat: distribute table in frontend

* test: distribute read demo

* test: distribute read demo

* test: distribute read demo

* add write spliter

* fix: node id changed to u64

* feat: datanode uses remote catalog implementation

* dist insert integrate table

* feat: specify region ids on creating table (#359)

* fix: compiling issues

* feat: datanode lease (#354)

* Some glue code about dist_insert

* fix: correctly wrap string value with quotes

* feat: create route

* feat: frontend catalog (#362)

* feat: integrate catalog to frontend

* feat: preserve partition rule on create

* fix: print tables on start

* chore: log in create route

* test: distribute read demo

* feat: support metasrv addr command line options

* feat: optimize DataNodeInstance creation (#368)

* chore: remove unnecessary changes

* chore: revert changes to src/api

* chore: revert changes to src/datanode/src/server.rs

* chore: remove opendal backend

* chore: optimize imports

* chore: revert changes to instance and region ids

* refactor: MetaKvBackend range

* fix: remove some wrap

* refactor: initiation of catalog

* feat: add region id to create table request and add heartbeat task to datanode instance

* fix: fix auto reconnect for heartbeat task

* chore: change TableValue::region_numbers to vec<u32>.

* fix: some tests

* fix: avoid concurrently start Heartbeat task by compare_exchange

* feat: refactor catalog key and values, separate table info into two kinds of keys

* feat: bump table id from metasrv

* fix: compare and set table id

* chore: merge develop

* fix: use integer serialization instead of string serialization

Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2022-11-08 20:40:16 +08:00
dennis zhuang
dd488e8d21 feat: adds from_unixtime function (#420) 2022-11-08 18:22:00 +08:00
fys
857054f70d feat: impl insert for DistTable (#406)
* feat: impl insert for dist_table in frontend

* add the logic of decode region id in datanode.
2022-11-08 17:19:17 +08:00
LFC
f34a99ff5a feat: use regex to filter out not supported MySQL stmt (#396)
* feat: use regex to filter out not supported MySQL stmt

* fix: resolve PR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-08 11:09:46 +08:00
Lei, Huang
2137587091 feat: datanode heartbeat (#377)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: databode lease

* feat: remote catalog (#356)

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: datanode heartbeat (#355)

* feat: add heartbeat task to instance

* feat: add node_id datanode opts

* fix: use real node id in heartbeat and meta client

* feat: distribute table in frontend

* test: distribute read demo

* test: distribute read demo

* test: distribute read demo

* add write spliter

* fix: node id changed to u64

* feat: datanode uses remote catalog implementation

* dist insert integrate table

* feat: specify region ids on creating table (#359)

* fix: compiling issues

* feat: datanode lease (#354)

* Some glue code about dist_insert

* fix: correctly wrap string value with quotes

* feat: create route

* feat: frontend catalog (#362)

* feat: integrate catalog to frontend

* feat: preserve partition rule on create

* fix: print tables on start

* chore: log in create route

* test: distribute read demo

* feat: support metasrv addr command line options

* feat: optimize DataNodeInstance creation (#368)

* chore: remove unnecessary changes

* chore: revert changes to src/api

* chore: revert changes to src/datanode/src/server.rs

* chore: remove opendal backend

* chore: optimize imports

* chore: revert changes to instance and region ids

* refactor: MetaKvBackend range

* fix: remove some wrap

* refactor: initiation of catalog

* feat: add region id to create table request and add heartbeat task to datanode instance

* fix: fix auto reconnect for heartbeat task

* chore: change TableValue::region_numbers to vec<u32>.

* fix: some tests

* fix: avoid concurrently start Heartbeat task by compare_exchange

* fix: some cr comments

* fix: fix unit tests

Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2022-11-07 17:10:43 +08:00
fys
c2e1b0857c refactor: optimize channel_manager (#401)
* refactor: use dashmap in channel manager

* add benchmark for channel manager

* access field in channel use AtomicUsize

* cr
2022-11-07 16:09:01 +08:00
LFC
16500b045b feat: distribute table in frontend (#328)
Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-05 10:41:59 +08:00
Ruihang Xia
bfcd74fd16 feat: benchmark suit based on nyc taxi dataset (#384)
* solve dep conflict

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: nyc taxi dataset writer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix some literals

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add some queries

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix progress bar

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* able to skip write or read

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename to nyc-taxi.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove main.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adapt new client api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* allow stdout output in this cli

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* some default values

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-11-04 14:13:17 +08:00
Lei, Huang
db2b577628 feat: remote catalog (#315)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: databode lease

* feat: remote catalog (#356)

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: datanode heartbeat (#355)

* feat: add heartbeat task to instance

* feat: add node_id datanode opts

* fix: use real node id in heartbeat and meta client

* feat: distribute table in frontend

* test: distribute read demo

* test: distribute read demo

* test: distribute read demo

* add write spliter

* fix: node id changed to u64

* feat: datanode uses remote catalog implementation

* dist insert integrate table

* feat: specify region ids on creating table (#359)

* fix: compiling issues

* feat: datanode lease (#354)

* Some glue code about dist_insert

* fix: correctly wrap string value with quotes

* feat: create route

* feat: frontend catalog (#362)

* feat: integrate catalog to frontend

* feat: preserve partition rule on create

* fix: print tables on start

* chore: log in create route

* test: distribute read demo

* feat: support metasrv addr command line options

* feat: optimize DataNodeInstance creation (#368)

* chore: remove unnecessary changes

* chore: revert changes to src/api

* chore: revert changes to src/datanode/src/server.rs

* chore: remove opendal backend

* chore: optimize imports

* chore: revert changes to instance and region ids

* refactor: MetaKvBackend range

* fix: remove some wrap

* refactor: initiation of catalog

* fix: next range request start key

* fix: mock delete range

* refactor: simplify range response handling

Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2022-11-04 11:43:31 +08:00
Jiachun Feng
e19b63f4f5 chore: meta mock test (#379)
* chore: meta mock

* chore: refacor datanode selector

* chore: create route mock test

* chore: add mock module

* chore: memory store for test

* chore: mock meta for test

* chore: ensure memorysotre has the same behavious with etcd

* chore: replace tokio lock to parking_lot
2022-11-03 18:33:29 +08:00
fys
1fad67cf4d feat: grpc client support multi peers (#380)
* feat: grpc client use channel manager

* cr
2022-11-03 11:55:22 +08:00
Jiachun Feng
2d4a44414d feat: refactor for test (#375)
* chore: add set_header macro & remove some unnessary code

* chore: channel_mannager with connector
2022-11-01 17:34:54 +08:00
LFC
ea2ebc0e87 feat: range partition rule (#304)
* feat: range partitioning rule

Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-01 16:09:23 +08:00
Jiachun Feng
dacfd12b8f feat: router impl (#363)
* feat: heartbeat lease & route api

* feat: batchput&cas

* chore: demo&ut

* chore: by cr

* chore: datanode selector

* chore: rename with_key_range to with_range

* chore: ut
2022-11-01 11:45:05 +08:00
Ning Sun
518b665f1e feat: Improve http sql api and attempt to add openapi docs (#361)
This patch changes output for our http SQL API and prepare it for our SQL editor development. Changes includes:

- includes aide for OAS 3.1 openapi documents, available at /v1/private/api.json
- simplified some of http handlers return type, use string or json directly
- created new HttpRecordsOutput type to hide internals of RecordBatch from end-user. It also tuned data structure to be friendly for application to consume
-  updated response struct to use code for success or detailed error code

Residual issue #366 

* feat: allow http post for our sql http api

* feat: update our http api and attempt to add openapi spec support

* test: correct test against new handler apis

* refactor: rename rows to records

* refactor: removed HttpResponse completely

* feat: add information to our openapi docs

* feat: add docs for sql interface response

* refactor: use struct to represent query so we can doc it via aide

* refactor: use arc wrapped api

* feat: add redoc UI support

* Update src/servers/src/http.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/servers/src/http.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: address review comments

* test: update integration tests for new api output

* refactor: make prometheus http apis compatible with recent changes

* refactor: get schema from stream

* test: add test for recordbatch to json serialization

* test: add todo for a test to be fixed later

* Revert "test: add todo for a test to be fixed later"

This reverts commit a5a50c7afb.

* fix: Revert "refactor: get schema from stream"

This reverts commit 945b685556.

* chore: add todo for pending issue #366

* chore: remove fixed server url in openapi docs

* feat: include error_code in json response

* refactor: use code over success field in json response

Co-authored-by: LFC <bayinamine@gmail.com>
2022-10-31 16:20:03 +08:00
dennis zhuang
0604eb7509 feat: prometheus remote write and read (#346)
* feat: scaffold for prometheus protocol handler

* feat: impl remote write and read for prometheus

* chore: make label matchers working in remote reading

* chore: case senstive regexp matching for labers and tweak restful api

* test: prometheus test

* test: adds test for prometheus handler and http server

* fix: typo in comment

* refactor: move snappy_compress and snappy_decompress

* fix: by code review

* fix: collect_timeseries_ids

* fix: timestamp and value column's value may be null
2022-10-28 18:47:16 +08:00
fys
6d4c0ad5a3 feat: add writespliter (#345)
* Add writespliter

* Partition_rule use reference, not Arc
2022-10-27 10:57:34 +08:00
Lei, Huang
932b30d299 refactor: catalog crate (#331)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop
2022-10-26 10:50:39 +08:00
Ruihang Xia
7fe39e9187 feat: support quering with logical plan in gRPC layer (#344)
* impl logical exec & example

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test on upper api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add todo to prost dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sign the TODO

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-10-25 16:05:53 +08:00
LFC
2ca667cbdf refactor: make table scan return physical plan (#326)
* refactor: return PhysicalPlan in Table trait's scan method, to support partitioned execution in Frontend's distribute read

* refactor: pub use necessary DataFusion types

* refactor: replace old "PhysicalPlan" and its adapters

Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2022-10-25 11:34:53 +08:00
Ruihang Xia
8ab43b65ea feat: serialize/deserialize logical and execution plan via substrait (#317)
* fix: change Utf8Array indice type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor: remove unused sub-crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: impl for both Logical and Execution plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor: move test-util subcrate into table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test: table scan logical plan round trip

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* drop support of physical plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename trait fns to encode/decode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* address review comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-10-24 15:29:33 +08:00
Jiachun Feng
b650656ae3 chore: refactor meta protocol (#332)
* chore: refactor channel_config

* chore: refactor grpc protocol

* feat: heartbeat streams
2022-10-21 20:30:57 +08:00
Ruihang Xia
bc9a2df9bf refactor: move test-util subcrate into table (#334)
* refactor: move test-util subcrate into table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: clean comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move MockTableEngine into test-util

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-10-21 14:39:40 +08:00
Lei, Huang
2d52f19662 feat: add table info (#323)
* refactor: add table_info method for Table trait

* feat: add table_info method to Table trait

* test: add more unit test

* fix: impl table_info for SystemTable

* test: fix failing test
2022-10-20 12:23:44 +08:00
LFC
d5800d0b60 feat: parse partition syntax in "create table" (#298)
* feat: parse partition syntax in "create table"

* Update src/sql/src/parsers/create_parser.rs

Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
2022-10-20 10:43:15 +08:00
Ruihang Xia
fbea07ea83 chore: remove unused dependencies (#319)
* chore: remove unused dependences

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: recover some dev-deps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-10-19 14:08:54 +08:00
Jiachun Feng
d5b34f8917 feat: metasrv (#300)
* meta: meta api&client

* meta: heartbeat server init

* feat: kv store

* chore: grpc server

* chore: meta server bootstrap

* feat: heartbeat client

* feat: route for create table

* chore: a channel pool manager

* feat: route client

* feat: store client

* chore: meta_client example

* chore: change schema

* chore: unit test & by cr

* chore: refactor meta client

* chore: add unit test
2022-10-19 11:02:58 +08:00
Ning Sun
f243649971 refactor: Removed openssl from build requirement (#308)
* refactor:replace another axum-test-helper branch

* refactor: upgrade opendal  version

* refactor: use cursor for file buffer

* refactor:remove native-tls in mysql_async

* refactor: use async block and pipeline for newer opendal api

* chore: update Cargo.lock

* chore: update dependencies

* docs: removed openssl from build requirement

* fix: call close on pipe writer to flush reader for parquet streamer

* refactor: remove redundant return

* chore: use pinned revision for our forked mysql_async

* style: avoid wild-card import in test code

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

* style: use chained call for builder

Co-authored-by: liangxingjian <965662709@qq.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2022-10-17 19:29:17 +08:00
Ning Sun
178f8b64b5 fix: Update pgwire and fix buffer overflow issue (#293) 2022-09-29 17:58:03 +08:00
fys
fe8327fc78 feat: support write data via influxdb line protocol in frontend (#280)
* feat: support influxdb line protocol write
2022-09-29 17:08:08 +08:00
Lei, Huang
25078e821b feat: type rewrite optimizer (#272)
* feat: add type conversion optimizer

* feat: add expr rewrite logical plan optimizer

* chore: add some doc

* fix: unit test

* fix: time zone issue in unit tests

* chore: add more tests

* fix: some CR comments

* chore: rebase develop

* chore: fix unit tests

* fix: unit test use timestamp with time zone

* chore: add more tests
2022-09-28 13:56:13 +08:00
LFC
ca732d45f9 feat: opentsdb support (#274)
* feat: opentsdb support

* fix: tests

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* refactor: remove feature flags for opentsdb and pg

* fix: resolve CR comments

* fix: resolve CR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-26 15:47:43 +08:00
dennis zhuang
5f322ba16e feat: impl default constraint for column (#273)
* feat: impl default value for column in schema

* test: add test for column's default value

* refactor: rename ColumnDefaultValue to ColumnDefaultConstraint

* fix: timestamp column may be a constant vector

* fix: test_shutdown_pg_server

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* chore: use table_info directly

Co-authored-by: LFC <bayinamine@gmail.com>

* refactor: by CR comments

Co-authored-by: LFC <bayinamine@gmail.com>
2022-09-22 10:43:21 +08:00
Lei, Huang
35ba0868b5 feat: impl filter push down to parquet reader (#262)
* wip add predicate definition

* fix value move

* implement predicate and prune

* impl filter push down in chunk reader

* add more expr tests

* chore: rebase develop

* fix: unit test

* fix: field name/index lookup when building pruning stats

* chore: add some meaningless test

* fix: remove unnecessary extern crate

* fix: use datatypes::schema::SchemaRef
2022-09-21 11:47:55 +08:00
Ning Sun
e67b0eb259 feat: Initial support of postgresql wire protocol (#229)
* feat: initial commit of postgres protocol adapter

* initial commit of postgres server

* feat: use common_io runtime and correct testcase

* fix previous tests

* feat: adopt pgwire api changes and add support for text encoded data

* feat: initial integration with datanode

* test: add feature flag to test

* fix: resolve lint warnings

* feat: add postgres feature flags for datanode

* feat: add support for newly introduced timestamp type

* feat: adopt latest datanode changes

* fix: address clippy warning for flattern scenario

* fix: make clippy great again

* fix: address issues found in review

* chore: sort dependencies by name

* feat: adopt new Output api

* fix: return error on unsupported data types

* refactor: extract common code dealing with record batches

* fix: resolve clippy warnings

* test: adds some unit tests postgres handler

* test: correct test for cargo update

* fix: update query module name

* test: add assertion for error content
2022-09-15 21:39:05 +08:00
fys
8400f8dfd4 chore: move query::Output to common-query module (#259)
* chore: move query::Output to common-query module

* chore: remove “query” dependency in client module
2022-09-15 10:07:58 +08:00
evenyag
0dce8946d4 ci: Add ci cache (#256) 2022-09-14 16:06:59 +08:00
discord9
20dcaa6897 feat: interval& None value for prev&`next (#252)
* test: for builtin functions

* test: expect fail for `datetime()`

* feat: add `interval()` fn(WIP)

* feat: `interval()` fn in builtin(UNTEST)

* refactor: move `py_vec_obj_to_array` to util.rs

* style: fmt

* test: simple `interval()` cases

* test: `interval()` with `last()`&`first()`

* doc: `ts` param of `interval()`

* log: common_telemetry for logging in script crate

* doc: corrsponding test fn for each .ron file

* feat: change to`mpsc` for schedule_job

* test: schedule_job

* dep: rm rustpython dep in common-function

* refactor: mv `schedule_job` into `Script` trait

* test: change to use `interval` to sample datapoint

* feat: add gen_none_array for generate None Array

* feat: impl Missing value for `prev`&`next`

* test: `sum(prev(values))`

* doc: add comment for why not support Float16 in `prev()`

* feat: add `interval` in py side mock module

* style: cargo fmt

* refactor: according to comments

* refactor: extract `apply_interval_function`

* style: cargo fmt

* refactor: remove `schedule()`

* style: cargo fmt
2022-09-14 10:48:27 +08:00
LFC
ec99eb0cd0 feat: frontend instance (#238)
* feat: frontend instance

* no need to carry column length in `Column` proto

* add more tests

* rebase develop

* create a new variant with already provisioned RecordBatches in Output

* resolve code review comments

* new frontend instance does not connect datanode grpc

* add more tests

* add more tests

* rebase develop

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 17:10:22 +08:00
dennis zhuang
03169c4a04 feat: impl scripts table and /run-script restful api (#230)
* feat: impl scripts table and /execute restful api

* fix: test failures

* fix: test failures

* feat: impl /run_script API

* refactor: rename run_script api to run-script and test script manager

* fix: remove println

* refactor: error mod

* refactor: by CR comments

* feat: rebase develop and change timestamp/gmt_crated/gmt_modified type to timestamp

* refactor: use assert_eq instread of assert

* doc: fix comment in Script#execute function
2022-09-13 15:09:00 +08:00
LFC
64b6b2afe1 feat: procedure macro to help writing UDAF (#242)
* feat: procedure macro to help writing UDAF

* resolve code review comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 10:39:44 +08:00
Lei, Huang
9366e77407 feat: impl timestamp type, value and vectors (#226)
* wip: impl timestamp data type

* add timestamp vectors

* adapt to recent changes to vector module

* fix all unit test

* rebase develop

* fix slice

* change default time unit to millisecond

* add more tests

* fix some CR comments

* fix some CR comments

* fix clippy

* fix some cr comments

* fix some CR comments

* fix some CR comments

* remove time unit in LogicalTypeId::Timestamp
2022-09-09 11:43:30 +08:00