Commit Graph

41 Commits

Author SHA1 Message Date
Ruihang Xia
6fb413ae50 ci: add toml format linter (#706)
* chore: run taplo format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ci: add workflow to check toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rerun formatter with ident to 4 spaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update check command

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-12-05 20:03:10 +08:00
shuiyisong
8232015998 fix: cargo sort in pre-commit (#689) 2022-12-02 16:19:31 +08:00
LFC
6127706b5b feat: support "use" stmt part 1 (#672)
* feat: a bare sketch of session; support "use" in MySQL server; modify insertion and selection related codes in Datanode
2022-12-01 17:05:32 +08:00
Dongxu Wang
6d24f7ebb6 refactor: bump axum 0.6, use recommended way to nest routes (#668) 2022-11-30 20:04:33 +08:00
dennis zhuang
c09775d17f feat: adds metrics, tracing and retry layer to object-store (#621) 2022-11-23 11:40:03 +08:00
Lei, HUANG
c144a1b20e feat: impl alter table in distributed mode (#572) 2022-11-22 15:17:25 +08:00
Yingwen
22ae983280 refactor: Use re-exported arrow mod from datatypes crate (#571) 2022-11-18 18:38:07 +08:00
Yingwen
55f18b5a0b refactor: Rename table-engine to mito (#539)
* refactor: Rename table-engine to mito

* style: Format codes

* docs: Update mito engine comment

* docs: Explain what is mito in README
2022-11-16 18:19:29 +08:00
Ruihang Xia
7ba512980a chore: add APACHE-2.0 license header (#518)
* feat: add license checker workflow

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix existing header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* specify license for internal sub-crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rustfmt

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-11-15 18:05:46 +08:00
Lei, Huang
2d869e1e43 refactor: datanode starts frontend (#471)
* refactor: dependency, from frontend depends on datanode to datanode depends on frontend

* wip: start frontend in datanode

* wip: migrate create database to frontend

* wip: impl alter table

* fix: CR comments
2022-11-12 21:07:18 +08:00
Lei, Huang
952e646e1d refactor: move insert-to-create logic to a separate crate (#447) 2022-11-10 17:16:40 +08:00
Lei, Huang
2137587091 feat: datanode heartbeat (#377)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: databode lease

* feat: remote catalog (#356)

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: datanode heartbeat (#355)

* feat: add heartbeat task to instance

* feat: add node_id datanode opts

* fix: use real node id in heartbeat and meta client

* feat: distribute table in frontend

* test: distribute read demo

* test: distribute read demo

* test: distribute read demo

* add write spliter

* fix: node id changed to u64

* feat: datanode uses remote catalog implementation

* dist insert integrate table

* feat: specify region ids on creating table (#359)

* fix: compiling issues

* feat: datanode lease (#354)

* Some glue code about dist_insert

* fix: correctly wrap string value with quotes

* feat: create route

* feat: frontend catalog (#362)

* feat: integrate catalog to frontend

* feat: preserve partition rule on create

* fix: print tables on start

* chore: log in create route

* test: distribute read demo

* feat: support metasrv addr command line options

* feat: optimize DataNodeInstance creation (#368)

* chore: remove unnecessary changes

* chore: revert changes to src/api

* chore: revert changes to src/datanode/src/server.rs

* chore: remove opendal backend

* chore: optimize imports

* chore: revert changes to instance and region ids

* refactor: MetaKvBackend range

* fix: remove some wrap

* refactor: initiation of catalog

* feat: add region id to create table request and add heartbeat task to datanode instance

* fix: fix auto reconnect for heartbeat task

* chore: change TableValue::region_numbers to vec<u32>.

* fix: some tests

* fix: avoid concurrently start Heartbeat task by compare_exchange

* fix: some cr comments

* fix: fix unit tests

Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2022-11-07 17:10:43 +08:00
Lei, Huang
932b30d299 refactor: catalog crate (#331)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop
2022-10-26 10:50:39 +08:00
Ruihang Xia
7fe39e9187 feat: support quering with logical plan in gRPC layer (#344)
* impl logical exec & example

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test on upper api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add todo to prost dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sign the TODO

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-10-25 16:05:53 +08:00
Jiachun Feng
d5b34f8917 feat: metasrv (#300)
* meta: meta api&client

* meta: heartbeat server init

* feat: kv store

* chore: grpc server

* chore: meta server bootstrap

* feat: heartbeat client

* feat: route for create table

* chore: a channel pool manager

* feat: route client

* feat: store client

* chore: meta_client example

* chore: change schema

* chore: unit test & by cr

* chore: refactor meta client

* chore: add unit test
2022-10-19 11:02:58 +08:00
Ning Sun
f243649971 refactor: Removed openssl from build requirement (#308)
* refactor:replace another axum-test-helper branch

* refactor: upgrade opendal  version

* refactor: use cursor for file buffer

* refactor:remove native-tls in mysql_async

* refactor: use async block and pipeline for newer opendal api

* chore: update Cargo.lock

* chore: update dependencies

* docs: removed openssl from build requirement

* fix: call close on pipe writer to flush reader for parquet streamer

* refactor: remove redundant return

* chore: use pinned revision for our forked mysql_async

* style: avoid wild-card import in test code

* Apply suggestions from code review

Co-authored-by: Yingwen <realevenyag@gmail.com>

* style: use chained call for builder

Co-authored-by: liangxingjian <965662709@qq.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2022-10-17 19:29:17 +08:00
LFC
ca732d45f9 feat: opentsdb support (#274)
* feat: opentsdb support

* fix: tests

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* refactor: remove feature flags for opentsdb and pg

* fix: resolve CR comments

* fix: resolve CR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-26 15:47:43 +08:00
Ning Sun
e67b0eb259 feat: Initial support of postgresql wire protocol (#229)
* feat: initial commit of postgres protocol adapter

* initial commit of postgres server

* feat: use common_io runtime and correct testcase

* fix previous tests

* feat: adopt pgwire api changes and add support for text encoded data

* feat: initial integration with datanode

* test: add feature flag to test

* fix: resolve lint warnings

* feat: add postgres feature flags for datanode

* feat: add support for newly introduced timestamp type

* feat: adopt latest datanode changes

* fix: address clippy warning for flattern scenario

* fix: make clippy great again

* fix: address issues found in review

* chore: sort dependencies by name

* feat: adopt new Output api

* fix: return error on unsupported data types

* refactor: extract common code dealing with record batches

* fix: resolve clippy warnings

* test: adds some unit tests postgres handler

* test: correct test for cargo update

* fix: update query module name

* test: add assertion for error content
2022-09-15 21:39:05 +08:00
dennis zhuang
c8cb705d9e ci: pre-commit configuration and hooks (#261)
* feat: adds pre-commit config and hooks

* refactor: sort all Cargo.toml by cargo-sort

* ci: adds conventional-pre-commit hook to pre-commit

* fix: remove .pre-commit-hooks.yaml

* fix: readme

* Update .pre-commit-config.yaml

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>

* ci: move clippy hook to push stage

* docs: install pre-push github hook

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
2022-09-15 11:30:08 +08:00
fys
8400f8dfd4 chore: move query::Output to common-query module (#259)
* chore: move query::Output to common-query module

* chore: remove “query” dependency in client module
2022-09-15 10:07:58 +08:00
LFC
5e67301c00 feat: implement alter table (#218)
* feat: implement alter table

* Currently we have no plans to support altering the primary keys (maybe never), so removed the related codes.

* make `alter` a trait function in table

* address other CR comments

* cleanup

* rebase develop

* resolve code review comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-06 13:44:34 +08:00
LFC
119ff2fc2e feat: create table through GRPC interface (#224)
* feat: create table through GRPC interface

* move `CreateExpr` `oneof` expr of `AdminExpr` in `admin.proto`, and implement the admin GRPC interface

* add `table_options` and `partition_options` to `CreateExpr`

* resolve code review comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-06 12:51:07 +08:00
evenyag
97d2aa4bfd feat: script engine and python impl (#219)
* feat: improve try_into_vector function

* Impl python mod and PyVector to execute script

* add AsSeq(BUT not IMPL)

* add&test pythonic_index, add into_py_obj(UNTEST)

* add into_datatypes_value(UNTEST)

* inplace setitem_by_index unsupport

* still struggle with testing AsSeq

* actually pyimpl AsSeq&AsMap

* add slice for PyVector

* improve visualibility for testing

* adjust for clippy

* add assert for test_execute_script

* add type anno in test

* feat: basic support for PyVector's operator with scalar (#64)

* feat: memory size of vector (#53)

* feat: improve try_into_vector function

* feat: impl memory_size function for vectors

* fix: forgot memory_size assertion in null vector test

* feat: use LargeUtf8 instead of utf8 for string, and rename LargeBianryArray to BinaryArray

* feat: memory_size only calculates heap size

* feat: impl bytes_allocated for memtable (#55)

* add init and constr

* rename type cast and add test

* fix bug in pyobj_to_val

* add default cast when no type specifed

* add basic add/sub/mul for array and scalar(value)

* cargo clippy

* comment out some println

* stricter clippy

* style: cargo fmt

* fix: string&bool support in val2pyobj & back

* style: remove println in test

* style: rm println in test mod in python.rs

* refactor: use wrap_index instead of pythonic_index

* refactor: right op in scalar_arith_op

* fix: stronger type& better test

* style: remove println

* fix: scalar sign/unsigned cast

* feat: improve try_into_vector function

* Impl python mod and PyVector to execute script

* add AsSeq(BUT not IMPL)

* add&test pythonic_index, add into_py_obj(UNTEST)

* add into_datatypes_value(UNTEST)

* inplace setitem_by_index unsupport

* still struggle with testing AsSeq

* actually pyimpl AsSeq&AsMap

* add slice for PyVector

* improve visualibility for testing

* adjust for clippy

* add assert for test_execute_script

* add type anno in test

* add init and constr

* rename type cast and add test

* fix bug in pyobj_to_val

* add default cast when no type specifed

* add basic add/sub/mul for array and scalar(value)

* cargo clippy

* comment out some println

* stricter clippy

* style: cargo fmt

* fix: string&bool support in val2pyobj & back

* style: remove println in test

* style: rm println in test mod in python.rs

* refactor: use wrap_index instead of pythonic_index

* refactor: right op in scalar_arith_op

* fix: stronger type& better test

* style: remove println

* fix: scalar sign/unsigned cast

* style: remove instead of comment out

* style: remove more comment out

* feat: support scalar div vector

* style: cargo fmt

* style: typo

* refactor: rename to correct var name

* refactor: directly use arrow2::array

* refactor: mv rsub&rdiv's op into a function

* test: add python expr test

* test: add test for PyList

* refactor: tweak order of arithmetics in rtruediv

* style: remove some `use`

* refactor: move `is_instance` to mod

* refactor: move fn to mod& move `use` to head

* style: cargo fmt

* fix: correct signed/unsigned cast

* refactor: wrap err msg in another fn

* style: cargo fmt

* style: remove ok_or_else for readability

* feat: add coprocessor fn(not yet impl)

* refactor: change back to wrapped_at

* fix: update Cargo.lock

* fix: update rustc version

* Update Rust Toolchain to nightly-2022-07-14

* feat: derive Eq when possible

* style: use `from` to avoid `needless_borrow` lint

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: python coprocessor with type annotation (#96)

* feat: add coprocessor fn

Signed-off-by: discord9 <zglzy29yzdk@gmail.com>

* feat: cast args into PyVector

* feat: uncomplete coprocessor

* feat: erase decorator in python ast

* feat: strip decorator in ast

* fix: change parse to `Interactive`

* style: format Cargo.toml

* feat: make coprocessor actually work

* feat: move coprocessor fn out of test mod

* feat: add error handling

* style: add some comment

* feat: rm type annotation

* feat: add type annotation support

* style: move compile method to vm closure

* feat: annotation for nullable

* feat: type coercion cast in annotation

* feat: actually cast(NOT TESTED)

* fix: allow single into(type)

* refactor: extract parse_type from parser

* style: cargo fmt

* feat: change to Expr to preserve location info

* feat: add CoprParse to deal parse check error

* style: add type anno doc for coprocessor

* test: add some test

* feat: add underscore as any type in annotation

* test: add parse& runtime testcases

* style: rm dbg! remnant

* style: cargo fmt

* feat: add more error prompt info

* style: cargo fmt

* style: add doc tests' missing `use`

* fix: doc test for coprocessor

* style: cargo fmt

* fix: add missing `use` for `cargo test --doc`

* refactor: according to reviews

* refactor: more tweaks according to reviews

* refactor: merge match arm

* refactor: move into different files(UNCOMPLELTE)

* refactor: split parse_copr into more function

* refactor: split `exec_coprocessor` to more fn

* style: cargo fmt

* feat: print Py Exceptions in String

* feat: error handling conform standards

* test: fix test_coprocessor

* feat: remove `into` in python

* test: remove all `into` in python test

* style: update comment

* refactor: move strip compile fn to impl Copr

* refactor: move `gen_schema` to impl copr

* refactor: move `check_cast_type` to impl copr

* refactor: if let to match

* style: cargo fmt

* refactor: better parse of keyword arg list

* style: cargo fmt

* refactor: some error handling(UNCOMPLETE)

* refactor: error handling to general Error type

* refactor: rm some Vec::new()

* test: modify all tests to ok

* style: reorder item

* refactor: fetch using iter

* style: cargo fmt

* style: fmt macro by hand

* refactor: rename InnerError to Error

* test: use ron to write test

* test: add test for exec_copr

* refactor: add parse_bin_op

* feat: add check_anno

* refactor: add some checker function

* refactor: exec_copr into smaller func

* style: add some comment

* refactor: add check for bin_op

* refactor: rm useless Result

* style: add pretty print for error with location

* feat: more info for pretty print

* refactor: mv pretty print to error.rs

* refactor: rm execute_script

* feat: add pretty print

* feat: add constant column support

* test: add test for constant column

* feat: add pretty print exec fn

* style: cargo fmt

* feat: add macro to chain call `.fail()`

* style: update doc for constant columns

* style: add lint to allow print in test fn

* style: cargo fmt

* docs: update some comment

* fix: ignore doctest for now

* refactor: check_bin_op

* refactor: parse_in_op, check ret anno fn

* refactor: rm check_decorator

* doc: loc add newline explain

* style: cargo fmt

* refactor: use Helper::try_into_vec in try_into_vec

* style: cargo fmt

* test: add ret anno test

* style: cargo fmt

* test: add name for .ron tests for better debug

* test: print emoji in test

* style: rm some comment out line

* style: rename `into` to `try_into` fn

* style: cargo fmt

* refactor: rm unuse serialize derive

* fix: pretty print out of bound fix

* fix: rm some space in pretty print

* style: cargo fmt

* test: not even a python fn def

* style: cargo fmt

* fix: pretty print off by one space

* fix: allow `eprint` in clippy lint

* fix: compile error after rebase develop

* feat: port 35 functions from DataFusion to Python Coprocessor (#137)

* refactor: `cargo clippy`

* feat: create a module

* style: cargo fmt

* feat: bind `pow()` function(UNTEST)

* test: add test for udf mod

* style: allow part eq not eq for gen code

* style: allow print in test lint

* feat: use PyObjectRef to handle more types

* feat: add cargo feature for udf modules

* style: rename feature to udf-builtins

* refactor: move away from mod.rs

* feat: add all_to_f64 cast fn

* feat: add bind_math_fn macro

* feat: add all simple math UDF

* feat: add `random(len)` math fn

* feat: port `avg()` from datafusion

* refactor: add `eval_aggr_fn`

* feat: add bind_aggr_fn macro

* doc: add comment for args of macro

* feat: add all UDAF from datafusion

* refactor: extract test to separate file

* style: cargo fmt

* test: add incomplete test

* test: add .ron test fn

* feat: support scalar::list

* doc: add comments

* style: rename VagueFloat/Int to LenFloat/IntVec

* test: for all fn(expect approx_median)

* test: better print

* doc: add comment for FloatWithError

* refactor: move test.rs out of builtins/

* style: cargo fmt

* doc: add comment for .ron file

* doc: update some comments

* test: EPS=1e-12 for float eq

* test: use f64::EPSILON instead

* test: change to 2*EPS

* test: cache interpreter for fast testing

* doc: remove a TODO which is done

* test: refacto to_py_obj fn

* fix: pow fn

* doc: add a TODO for type_.rs

* test: use new_int/float in test serde

* test: for str case

* style: cargo fmt

* feat: cast PyList to ScalarValue::List

* test: cast scalar to py obj and back

* feat: cast to PyList

* test: cast from PyList

* test: nested PyVector unsupported

* doc: remove unrunable doctest

* test: replace PartialEq with impl just_as_expect

* doc: add name for discord9's TODO

* refactor: cahnge to vm.ctx.new_** instead

* doc: complete a TODO

* refactor: is_instance and other minor problem

* refactor: remove type_::is_instance

* style: cargo fmt

* feat: rename to `greptime_builtin`

* fix: error handling for PyList datatype

* style: fix clippy warning

* test: for PyList

* feat: Python Coprocessor MVP (#180)

* feat: add get_arrow_op

* feat: add comparsion op(UNTESTED)

* doc: explain why no rich compare

* refactor: py_str2str&parse_keywords

* feat: add DecoratorArgs

* refactor: parse_keywords ret Deco Args

* style: remove unused

* doc: add todo

* style: remove some unused fn

* doc: add comment for copr's field

* feat: add copr_engine module

* refactor: move to `script` crate

* style: clean up cargo.toml

* feat: add query engine for copr engine

* refactor: deco args into separate struct

* test: update corrsponding test

* feat: async coprocessor engine

* refactor: add `exec_parsed` fn

* feat: sync version of coprocessor(UNTEST)

* refactor: remove useless lifetime

* feat: new type for async stream record batch

* merge: from PR#137 add py builtins

* toolchain: update rustc to nightly-08-16

* feat: add `exec_with_cached_vm` fn(Can't compile)

* toolchain: revert to 07-14

* fix: `exec_with_cached_vm`

* fix: allow vector[_] in params

* style: cargo fmt

* doc: update comment on `_`&`_|None`

* fix: allow import&ignore type anno is ok

* feat: allow ignore return types

* refsctor: remove unused py files in functions/

* style: fmt&clippy

* refactor: python modules (#186)

* refactor: move common/script to script

* fix: clippy warnings and refactor python modules

* refactor: remove modules mod rename tests mod

* feat: adds Script and ScriptEngine trait, then impl PyScript/PyScriptEngine

* refactor: remove pub use some functions in script

* refactor: python error mod

* refactor: coprocessor and vector

* feat: adds engine test and greptime.vector function to create vector from iterable

* fix: adds a blank line to cargo file end

* fix: compile error after rebase develop

* feat: script endpoint for http server (#206)

* feat: impl /scripts API for http server

* feat: adds http api version

* test: add test for scripts handler and endpoint

* feat: python side mock module and more builtin functions (#209)

* feat: add python side module(for both mock and real upload script)

* style: add *.pyc to gitignore

* feat: move copr decorator(in .py) to greptime.py

* doc: update comment for `datetime`&`mock_tester`&gitignore

* feat: `filter()` a array with bool array(UNTESTED)

* feat: `prev()`ious elem in array ret as new array(UNTEST)

* feat: `datetime()` parse date time string and ret integer(UNTEST)

* fix: add missing return&fmt

* fix: allow f32 cast to PyFloat

* fix: `datetime()`'s last token now parsed

* test: `calc_rvs` now can run with builtin module

* feat: allow rich compare which ret bool array

* feat: logic and(`&`) for bool array

* style: cargo fmt

* feat: index PyVector by bool array

* feat: alias `ln` as `log` in builtin modules

* feat: logic or(`|`)&not( `~`) for bool array

* feat: add `post` for @copr in py side mod

* feat: change datetime return to i64

* feat: py side mod `post` script to given address

* fix: add `engine` field in `post` in py side mod

* refactor: use `ConstantVector` in `pow()` builtin

* fix: prev ret err for zero array

* doc: rm comment out code

* test: incomplete pyside mod test case

* git: ignore all __pycache__

* style: fmt&clippy

* refactor: split py side module into exmaple&gptime

* feat: init_table in py using `v1/sql`  api

* feat: calc_rvs now run both locally and remote

* doc: add doc for how to run it

* fix: comment out start server code in test

* fix: clippy warnings

* fix: http test url

* fix: some CR problems

* fix: some CR problems

* refactor: script executor for instance

* refactor: remove engine param in execute_script

* chore: Remove unnecessary allow attributes

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Discord9 <discord9@163.com>
Co-authored-by: discord9 <zglzy29yzdk@gmail.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
2022-09-01 20:38:39 +08:00
fys
db55c69117 feat: impl grpc physical plan (#212)
* chore: rename "convert.rs" to "serde.rs"

* proto definition

* impl "projection"

* add mock_input_exec for test

* impl physical plan execution
2022-08-31 21:43:50 +08:00
Lei, Huang
2373d676f7 feat: add Date type and value (#189)
* wip: add Date type and value

* fix some cr comments

* impl Date values

* finish date type

* optimize Date value serialization

* add some tests

* fix some cr comments

* add some more test
2022-08-23 18:04:32 +08:00
LFC
ccda17248e feat: unify servers and mysql server in datanode (#172)
* address PR comments

address PR comments

use 3306 for mysql server's default port

upgrade metric to version 0.20

move crate "servers" out of "common"

make mysql io threads count configurable in config file

add snafu backtrace for errors with source

use common-server error for mysql server

add test for grpc server

refactor testing codes

fix rustfmt check

start mysql server in datanode

move grpc server codes from datanode to common-servers

feat: unify servers

* rebase develop and resolve conflicts

* remove an unnecessary todo

Co-authored-by: luofucong <luofucong@greptime.com>
2022-08-17 14:29:12 +08:00
Lei, Huang
a1c4921933 feat: impl create table sql execution (#168)
* catalog manager allocates table id

* rebase develop

* add some tests

* add some more test

* fix some cr comments

* insert into system catalog

* use slice pattern to simplify code

* add optional dependencies

* add sql-to-request test

* successfully recover

* fix unit tests

* rebase develop

* add some tests

* fix some cr comments

* fix some cr comments

* add a lock to CatalogManager

* feat: add gmt_created and gmt_modified columns to system catalog table
2022-08-17 10:53:19 +08:00
fys
34133fae5a feat: impl select (grpc) (#138)
* SelectExpr: change to oneof expr

* Convert between Vec<u8> and SelectResult

* Chore: use encode_to_vec and decode, instead of encode_length_delimited_to_vec and decode_length_delimited

* Chore: move bitset into separate file

* Grpc select impl
2022-08-15 18:31:47 +08:00
dennis zhuang
41ffbe82f8 feat: impl table manifest (#157)
* feat: impl TableManifest and refactor table engine, object store etc.

* feat: persist table metadata when creating it

* fix: remove unused file src/storage/src/manifest/impl.rs

* feat: impl recover table info from manifest

* test: add open table test and table manifest test

* fix: resolve CR problems

* fix: compile error and remove region id

* doc: describe parent_dir

* fix: address CR problems

* fix: typo

* Revert "fix: compile error and remove region id"

This reverts commit c14c250f8a.

* fix: compile error and generate region id by table_id and region number
2022-08-12 10:47:33 +08:00
Ning Sun
97be052b33 feat: update tonic/prost and simplify build requirements (#130)
* feat: update tonic/prost and simplify build requirements

* doc: update readme for protoc installtion
2022-08-04 23:11:39 +08:00
Lei, Huang
7395920bc8 move catalog-related traits and struct to a catalog crate (#134) 2022-08-04 11:05:28 +08:00
evenyag
bf5975ca3e feat: Prototype of the storage engine (#107)
* feat: memtable flush (#63)

* wip: memtable flush

* optimize schema conversion

* remove unnecessary import

* add parquet file verfication

* add backtrace to error

* chore: upgrade opendal to 0.9 and fixed some problems

* rename error

* fix: error description

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>

* feat: region manifest service (#57)

* feat: adds Manifest API

* feat: impl region manifest service

* refactor: by CR comments

* fix: storage error mod test

* fix: tweak storage cargo

* fix: tweak storage cargo

* refactor: by CR comments

* refactor: rename current_version

* feat: add wal writer (#60)

* feat: add Wal

* upgrade engine for wal

* fix: unit test for wal

* feat: wal into region

* fix: unix test

* fix clippy

* chore: by cr

* chore: by cr

* chore: prevent test data polution

* chore: by cr

* minor fix

* chore: by cr

* feat: Implement flush (#65)

* feat: Flush framework

- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports

refactor: Use ParquetWriter to replace FlushTask

refactor: FsAccessLayer takes object store as param

chore: Remove todo from doc comments

feat: Move wal to WriterContext

chore: Fix clippy

chore: Add backtrace to WriteWal error

* feat: adds manifest to region and refactor sst/manifest dir config (#72)

* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig

* refactor: ensure path ends with '/' in ManifestLogStorage

* fix: style

* refactor: normalize storage directory path and minor changes by CR

* refactor: doesn't need slash any more

* feat: Implement apply_edit() and add timestamp index to schema (#73)

* feat: Implement VersionControl::apply_edit()

* feat: Add timestamp index to schema

* feat: Implement Schema::timestamp_column()

* feat: persist region metadata to manifest (#74)

* feat: persist metadata when creating region or sst files

* fix: revert FileMeta comment

* feat: resolve todo

* fix: clippy warning

* fix: revert files_to_remove type in RegionEdit

* feat: impl SizeBasedStrategy for flush (#76)

* feat: impl SizeBasedStrategy for flush

* doc: get_mutable_limitation

* fix: code style and comment

* feat: align timestamp (#75)

* feat: align timestamps in write batch

* fix cr comments

* fix timestamp overflow

* simplify overflow check

* fix cr comments

* fix clippy issues

* test: Fix region tests (comment out some unsupported tests) (#82)

* feat: flush job (#80)

* feat: flush job

* fix cr comments

* move file name instead of clone

* comment log file test (#84)

* feat: improve MemtableVersion (#78)

* feat: improve MemtableVersion

* feat: remove flushed immutable memtables and test MemtableVersion

* refactor: by CR comments

* refactor: clone kv in iterator

* fix: clippy warning

* refactor: Make BatchIterator supertrait of Iterator (#85)

* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83)

* feat: Insert multiple memtables by time range (#77)

* feat: memtable::Inserter supports insert multiple memtables by time range

* chore: Update timestamp comment

* test: Add tests for Inserter

* test: Fix region tests (comment out some unsupported tests)

* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()

* chore: rename aligned_by_bucket to align_by_bucket

* fix: Fix compile errors

* fix: sst and manifest dir (#86)

* Set RowKeyDescriptor::enable_version_column to false by default

* feat: Implement write stall (#90)

* feat: Implement write stall

* chore: Update comments

* feat: Support reading multiple memtables (#93)

* feat: Support reading multiple memtables

* test: uncomment tests rely on snapshot read

* feat: wal format (#70)

* feat: wal codec

* chore: minor fix

* chore: comment

* chore: by cr

* chore: write_batch_codec mod

* chore: by cr

* chore: upgrade proto

* chore: by cr

* fix failing test

* fix failing test

* feat: manifest to wal (#100)

* feat: write manifest to wal

* chore: sequence into wal

* chore: by cr

* chore: by cr

* refactor: create log store (#104)

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: fariygirl <clickmetoday@163.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: Fix clippy

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
Co-authored-by: fariygirl <clickmetoday@163.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
2022-07-25 15:26:00 +08:00
Jiachun Feng
6cf1da35ee feat: add grpc impl (#50)
* feat: add grpc impl

* feat: add grpc server

* some ut

* verson format: a.b

* code style

* admin request/response

* by cr

* admin api

* by cr

* chore: by cr

* chore: by cr
2022-07-06 20:56:16 +08:00
dennis zhuang
e78c015fc0 TableEngine and SqlHandler impl (#45)
* Impl TableEngine, bridge to storage

* Impl sql handler to process insert sql

* fix: minor changes and typo

* test: add datanode test

* test: add table-engine test

* fix: code style

* refactor: split out insert mod from sql and minor changes by CR

* refactor: replace with_context with context
2022-06-17 11:36:49 +08:00
dennis zhuang
93cbdbee9a feat: adds common-telemetry crate and adds logs/metrics to query engine (#25)
* feat: adds common-telemetry crate and logging mod

* refactor: common telemetry

* feat: adds metric mod and exports metrics by http

* feat: adds metrics to query engine and datanodes

* refactor: rename metrics

* refactor: rename ElapsedTimer struct and elapsed_timer macro

* refactor: log error when fail to new data node.

* fix: use backtrace crate instead of unstable feature

* feat: add must_use attr to Timer

* fix: only reserve Debug attribute for Timer
2022-05-12 17:42:48 +08:00
dennis zhuang
df1a99e27f feat: test http server api (#21)
* feat: adds http server test

* feat: limit sql output to 10 rows
2022-05-09 14:54:03 +08:00
evenyag
63d9aa1bff refactor: Refactor datanode error and impl ErrorExt for it 2022-05-07 11:34:23 +08:00
evenyag
aa1a6b2205 feat: Add common-error crate and implement opaque error type. 2022-05-06 20:04:14 +08:00
dennis zhuang
f0db878bf5 feat: http server and cmd crate etc. (#15)
* feat: adds cmd crate and http server

* feat: impl sql http handler

* feat: convert all arrow array types

* feat: adds query test

* feat: adds test for datanode

* fix: format

* feat: refactor state.rs

* feat: adds collect test

* fix: by code review

* fix: style
2022-05-06 17:34:29 +08:00
evenyag
1fa0b4e3f9 chore: Setup code skeleton of datanode 2022-04-21 11:40:46 +08:00
evenyag
ced9a7c97f chore: Create datanode, datatypes, log-store, object-store, query, table crates 2022-04-20 16:34:22 +08:00