Commit Graph

41 Commits

Author SHA1 Message Date
Lei, HUANG
7bbc679c76 fix: revert script dependenciex 2023-01-06 15:15:41 +08:00
Lei, HUANG
0b3a2cbcda fix: revert cargo workspace dependencies 2023-01-06 15:10:04 +08:00
Lei, HUANG
53ee85cdad feat: use raft-engine crate to reimplement logstore (#799)
(cherry picked from commit 8f5ecefc90)
2023-01-06 15:05:55 +08:00
discord9
e85780b5e4 refactor: rename some mod.rs to <MOD_NAME>.rs (#784)
* refactor: rename `mod.rs` to <MOD_NAME>.rs

* refactor: not rename mod.rs in benches/
2022-12-26 12:48:34 +08:00
LFC
dc52a51576 chore: upgrade to Arrow 29.0 and use workspace package and dependencies (#782)
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies

* fix: resolve PR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-12-23 14:28:37 +08:00
LFC
ea9af42091 chore: upgrade Rust to nightly 2022-12-20 (#772)
* chore: upgrade Rust to nightly 2022-12-20

* chore: upgrade Rust to nightly 2022-12-20

Co-authored-by: luofucong <luofucong@greptime.com>
2022-12-21 19:32:30 +08:00
LFC
77182f5024 chore: upgrade Arrow to version 28, and DataFusion to 15 (#771)
Co-authored-by: luofucong <luofucong@greptime.com>
2022-12-21 17:02:11 +08:00
Lei, HUANG
0653301754 feat: replace arrow2 with official implementation 🎉 (#753)
* chore: kick off. change datafusion/arrow/parquet to target version

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: replace one last datafusion dep

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: arrow_array switch to arrow

* chore: update dep of binary vector

* chore: fix wrong merge commit

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: Switch to datatypes2

* feat: Make recordbatch compile

* chore: sort Cargo.toml

* feat: Fix common::recordbatch compiler errors

* feat: Fix recordbatch test compiling issue

* fix: api crate (#708)

* fix: rename ConcreteDataType::timestamp_millis_type to ConcreteDataType::timestamp_millisecond_type. fix other warnings regarding timestamp

* fix: revert changes in datatypes2

* fix: helper

* chore: delete datatypes based on arrow2

* feat: Fix some compiler errors in common::query (#710)

* feat: Fix some compiler errors in common::query

* feat: test_collect use vectors api

* fix: common-query subcrate (#712)

* fix: record batch adapter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix error enum

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: Fix common::query compiler errors (#713)

* feat: Move conversion to ScalarValue to value.rs

* fix: Fix common::query compiler errors

This commit also make InnerError pub(crate)

* feat: Implements diff accumulator using WrapperType (#715)

* feat: Remove usage of opaque error from common::recordbatch

* feat: Remove opaque error from common::query

* feat: Fix diff compiler errors

Now common_function just use common_query's Error and Result. Adds
a LargestType associated type to LogicalPrimitiveType to get the largest
type a logical primitive type can cast to.

* feat: Remove LargestType from NativeType trait

* chore: Update comments

* feat: Restrict Scalar::RefType of WrapperType to itself

Add trait bound `for<'a> Scalar<RefType<'a> = Self>` to WrapperType

* chore: Address CR comments

* chore: Format codes

* fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Revert "fix: fix compile error for mean/polyval/pow/interp ops"

This reverts commit fb0b4eb826.

* fix: Fix compiler errors in argmax/rate/median/norm_cdf (#716)

* fix: Fix compiler errors in argmax/rate/median/norm_cdf

* chore: Address CR comments

* fix: fix compile error for mean/polyval/pow/interp ops (#717)

* fix: fix compile error for mean/polyval/pow/interp ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify type bounds

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf errors (#718)

fix: fix argmin/percentile/clip/interp/scipy_stats_norm_pdf compiler errors

* fix: fix other compile error in common-function (#719)

* further fixing

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: Fix tests and clippy for common-function subcrate (#726)

* further fixing

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix all compile errors in common function

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert test changes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: row group pruning (#725)

* fix: row group pruning

* chore: use macro to simplify stats implemetation

* fxi: CR comments

* fix: row group metadata length mismatch

* fix: simplify code

* fix: Fix common::grpc compiler errors (#722)

* fix: Fix common::grpc compiler errors

This commit refactors RecordBatch and holds vectors in the RecordBatch
struct, so we don't need to cast the array to vector when doing
serialization or iterating the batch.

Now we use the vector API instead of the arrow API in grpc crate.

* chore: Address CR comments

* fix common record batch

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: Fix compile error in server subcrate (#727)

* fix: Fix compile error in server subcrate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused type alias

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* explicitly panic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/storage/src/sst/parquet.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: Fix common grpc expr (#730)

* fix compile errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename fn names

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix wranings in common-time

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: pre-cast to avoid tremendous match arms (#734)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: upgrade storage crate to arrow and parquet offcial impl (#738)

* fix: compile erros

* fix: parquet reader and writer

* fix: parquet reader and writer

* fix: WriteBatch IPC encode/decode

* fix: clippy errors in storage subcrate

* chore: remove suspicious unwrap

* fix: some cr comments

* fix: CR comments

* fix: CR comments

* fix: Fix compiler errors in catalog and mito crates (#742)

* fix: Fix compiler errors in mito

* fix: Fix compiler errors in catalog crate

* style: Fix clippy

* chore: Fix use

* Merge pull request #745

* fix nyc-taxi and util

* Merge branch 'replace-arrow2' into fix-others

* fix substrait

* fix warnings and error in test

* fix: Fix imports in optimizer.rs

* fix: errors in optimzer

* fix: remove unwrap

* fix: Fix compiler errors in query crate (#746)

* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* Update src/query/src/query_engine/state.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: frontend compile errors (#747)

fix: fix compile errors in frontend

* fix: Fix compiler errors in script crate (#749)

* fix: Fix compiler errors in state.rs

* fix: fix compiler errors in state

* feat: upgrade sqlparser to 0.26

* fix: fix datafusion engine compiler errors

* fix: Fix some tests in query crate

* fix: Fix all warnings in tests

* feat: Remove `Type` from timestamp's type name

* fix: fix query tests

Now datafusion already supports median, so this commit also remove the
median function

* style: Fix clippy

* feat: Remove RecordBatch::pretty_print

* chore: Address CR comments

* feat: Add column_by_name to RecordBatch

* feat: modify select_from_rb

* feat: Fix some compiler errors in vector.rs

* feat: Fix more compiler errors in vector.rs

* fix: fix table.rs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: Fix compiler errors in coprocessor

* fix: Fix some compiler errors

* fix: Fix compiler errors in script

* chore: Remove unused imports and format code

* test: disable interval tests

* test: Fix test_compile_execute test

* style: Fix clippy

* feat: Support interval

* feat: Add RecordBatch::columns and fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: Fix All The Tests! (#752)

* fix: Fix several tests compile errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: some compile errors in tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: compile errors in frontend tests

* fix: compile errors in frontend tests

* test: Fix tests in api and common-query

* test: Fix test in sql crate

* fix: resolve substrait error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: add more test

* test: Fix tests in servers

* fix instance_test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test: Fix tests in tests-integration

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>

* fix: clippy errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
2022-12-15 18:49:12 +08:00
dennis zhuang
7f59758e69 feat: bump opendal version to 0.22 (#721)
* feat: bump opendal version to 0.22

* fix: LoggingLayer
2022-12-08 11:19:21 +08:00
LFC
833216d317 refactor: directly invoke Datanode methods in standalone mode (part 1) (#694)
* refactor: directly invoke Datanode methods in standalone mode

* test: add more unit tests

* fix: get rid of `println` in testing codes

* fix: resolve PR comments

* fix: resolve PR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-12-07 11:37:59 +08:00
discord9
2034b40f33 chore: update RustPython dependence(With a tweaked fork) (#655)
* refactor: update RsPy

* depend: add `rustpython-pylib`

* feat: add_frozen stdlib for every vm init

* feat: limit stdlib to a selected few

* chore: use `rev` instead of branch` im depend

* refactor: rename to allow_list

* feat: use opt level one

* doc: add username for TODO&change optimize to 0

* style: fmt .toml
2022-12-06 14:15:00 +08:00
Ruihang Xia
6fb413ae50 ci: add toml format linter (#706)
* chore: run taplo format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ci: add workflow to check toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rerun formatter with ident to 4 spaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update check command

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-12-05 20:03:10 +08:00
dennis zhuang
6720bc5f7c fix: validate create table request in mito engine (#690)
* fix: validate create table request in mito engine

* fix: comment

* chore: remove TIMESTAMP_INDEX in system.rs
2022-12-05 11:01:43 +08:00
shuiyisong
8232015998 fix: cargo sort in pre-commit (#689) 2022-12-02 16:19:31 +08:00
LFC
6127706b5b feat: support "use" stmt part 1 (#672)
* feat: a bare sketch of session; support "use" in MySQL server; modify insertion and selection related codes in Datanode
2022-12-01 17:05:32 +08:00
dennis zhuang
fdc73fb52f perf: cache python interpreter in TLS (#649)
* perf: cache python interpreter when executing coprocessors

* test: speedup test_execute_script by reusing interpreter

* fix: remove comment

* chore: use get_or_insert_with instead
2022-11-29 14:41:37 +08:00
Dongxu Wang
b6fa316c65 chore: correct typos (#589) (#592) 2022-11-21 14:07:45 +08:00
Yingwen
22ae983280 refactor: Use re-exported arrow mod from datatypes crate (#571) 2022-11-18 18:38:07 +08:00
Yingwen
55f18b5a0b refactor: Rename table-engine to mito (#539)
* refactor: Rename table-engine to mito

* style: Format codes

* docs: Update mito engine comment

* docs: Explain what is mito in README
2022-11-16 18:19:29 +08:00
Ruihang Xia
7ba512980a chore: add APACHE-2.0 license header (#518)
* feat: add license checker workflow

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix existing header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* specify license for internal sub-crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rustfmt

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-11-15 18:05:46 +08:00
Ruihang Xia
1565c8d236 chore: specify import style in rustfmt (#460)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2022-11-15 15:58:54 +08:00
dennis zhuang
102e512a0a feat: enable freeze-stdlib feature in rust-python (#511) 2022-11-15 15:06:58 +08:00
LFC
d10e45f4aa feat: distributed query in Frontend (#486)
* feat: distributed query in Frontend

Co-authored-by: luofucong <luofucong@greptime.com>
2022-11-14 18:15:49 +08:00
dennis zhuang
74ea529d1a feat: move time index metadata from schema into field (#444)
* feat: move time index metadata from schema into field

* chore: remove useless code

* test: test select with column alias

* fix: conflicts with develop branch

* test: add test

* test: order by timestamp to ensure query results order

* fix: comment
2022-11-11 15:36:27 +08:00
dennis zhuang
e7b4d24df5 feat: create database (#451)
* feat: parsing create database statement

* feat: impl create database in datanode

* feat: supports insert into catalog.schema.table

* fix: conflicts with develop branch

* test: create database then insert and query

* fix: grpc schema provider

* feat: use CatalogManager::register_schema instead of CatalogProvide::register_schema

* refactor: revert InsertExpr catalog_name and schema_name

* fix: revert database.proto

* fix: revert client cargo

* feat: accepts schema.table as table name in sql

Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
2022-11-11 14:15:38 +08:00
dennis zhuang
e62b302fb2 feat: some improvements on python coprocessor (#423)
* feat: supports list array in arrow_array_get

* feat: supports string and list type conversions in python coprocessor

* test: add test cases for returning list in coprocessor
2022-11-10 11:53:27 +08:00
Lei, Huang
2137587091 feat: datanode heartbeat (#377)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: databode lease

* feat: remote catalog (#356)

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* refactor: merge refactor/catalog-crate

* feat: table key with version

* feat: impl KvBackend for MetaClient

* fix: integrate metaclient

* fix: catalog use local table info as baseline

* fix: sync metsrv

* fix: wip

* fix: update remote catalog on register and deregister

* refactor: CatalogProvider

* refactor: CatalogManager

* fix: catalog key filtering

* fix: pass some test

* refactor: catalog iterating

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop

* chore: merge catalog crate

* fix: adapt to recent meta-client api change

* feat: datanode heartbeat (#355)

* feat: add heartbeat task to instance

* feat: add node_id datanode opts

* fix: use real node id in heartbeat and meta client

* feat: distribute table in frontend

* test: distribute read demo

* test: distribute read demo

* test: distribute read demo

* add write spliter

* fix: node id changed to u64

* feat: datanode uses remote catalog implementation

* dist insert integrate table

* feat: specify region ids on creating table (#359)

* fix: compiling issues

* feat: datanode lease (#354)

* Some glue code about dist_insert

* fix: correctly wrap string value with quotes

* feat: create route

* feat: frontend catalog (#362)

* feat: integrate catalog to frontend

* feat: preserve partition rule on create

* fix: print tables on start

* chore: log in create route

* test: distribute read demo

* feat: support metasrv addr command line options

* feat: optimize DataNodeInstance creation (#368)

* chore: remove unnecessary changes

* chore: revert changes to src/api

* chore: revert changes to src/datanode/src/server.rs

* chore: remove opendal backend

* chore: optimize imports

* chore: revert changes to instance and region ids

* refactor: MetaKvBackend range

* fix: remove some wrap

* refactor: initiation of catalog

* feat: add region id to create table request and add heartbeat task to datanode instance

* fix: fix auto reconnect for heartbeat task

* chore: change TableValue::region_numbers to vec<u32>.

* fix: some tests

* fix: avoid concurrently start Heartbeat task by compare_exchange

* fix: some cr comments

* fix: fix unit tests

Co-authored-by: jiachun <jiachun_fjc@163.com>
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: fys <1113014250@qq.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
2022-11-07 17:10:43 +08:00
discord9
3d195ff858 feat: bind Greptime's own UDF&UDAF into Python Coprocessor Module (#335)
* feat: port own UDF&UDAF into py copr(untest yet)

* refactor: move UDF&UDAF to greptime_builtins

* feat: support List in val2py_obj

* test: some testcases for newly added UDFs

* test: complete test for all added gpdb's own UDF

* refactor: add underscore for long func name

* feat: better error message

* fix: typo
2022-11-04 15:49:41 +08:00
Yingwen
cba611b9f5 refactor: Serialize RawSchema/RawTableMeta/RawTableInfo (#382)
* refactor: Serialize Schema/TableMeta/TableInfo to raw structs

* test: Add tests for raw struct conversion

* style: Fix clippy

* refactor: SchemaBuilder::timestamp_index takes Option<usize>

So caller could chain the timestamp_index method call where there is no
timestamp index.

* style(datatypes): Chains SchemaBuilder method calls
2022-11-04 11:25:17 +08:00
Lei, Huang
932b30d299 refactor: catalog crate (#331)
* chore: refactor dir for local catalog manager

* refactor: CatalogProvider returns Result

* refactor: SchemaProvider returns Result

* feat: add kv operations to remote catalog

* chore: refactor some code

* feat: impl catalog initialization

* feat: add register table and register system table function

* refactor: add table_info method for Table trait

* chore: add some tests

* chore: add register schema test

* chore: fix build issue after rebase onto develop

* refactor: mock to separate file

* build: failed to compile

* fix: use a container struct to bridge KvBackend and Accessor trait

* feat: upgrade opendal to 0.17

* test: add more tests

* chore: add catalog name and schema name to table info

* chore: add catalog name and schema name to table info

* chore: rebase onto develop

* refactor: common-catalog crate

* refactor: remove remote catalog related files

* fix: compilation

* feat: add table version to TableKey

* feat: add node id to TableValue

* fix: some CR comments

* chore: change async fn create_expr_to_request to sync

* fix: add backtrace to errors

* fix: code style

* fix: CatalogManager::table also requires both catalog_name and schema_name

* chore: merge develop
2022-10-26 10:50:39 +08:00
Lei, Huang
2d52f19662 feat: add table info (#323)
* refactor: add table_info method for Table trait

* feat: add table_info method to Table trait

* test: add more unit test

* fix: impl table_info for SystemTable

* test: fix failing test
2022-10-20 12:23:44 +08:00
dennis zhuang
5f322ba16e feat: impl default constraint for column (#273)
* feat: impl default value for column in schema

* test: add test for column's default value

* refactor: rename ColumnDefaultValue to ColumnDefaultConstraint

* fix: timestamp column may be a constant vector

* fix: test_shutdown_pg_server

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* chore: use table_info directly

Co-authored-by: LFC <bayinamine@gmail.com>

* refactor: by CR comments

Co-authored-by: LFC <bayinamine@gmail.com>
2022-09-22 10:43:21 +08:00
dennis zhuang
dfa3012396 feat: improve python coprocesssor parser (#260)
* feat: supports DateTime, Date and Timestamp column type to be returned by py scripts

* feat: improve coprocessor compiler, make it work better

* fix: comments

* fix: typo

* Update src/script/src/python/vector.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/script/src/python/coprocessor.rs

Co-authored-by: LFC <bayinamine@gmail.com>

Co-authored-by: LFC <bayinamine@gmail.com>
2022-09-15 16:18:33 +08:00
dennis zhuang
c8cb705d9e ci: pre-commit configuration and hooks (#261)
* feat: adds pre-commit config and hooks

* refactor: sort all Cargo.toml by cargo-sort

* ci: adds conventional-pre-commit hook to pre-commit

* fix: remove .pre-commit-hooks.yaml

* fix: readme

* Update .pre-commit-config.yaml

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>

* ci: move clippy hook to push stage

* docs: install pre-push github hook

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
2022-09-15 11:30:08 +08:00
fys
8400f8dfd4 chore: move query::Output to common-query module (#259)
* chore: move query::Output to common-query module

* chore: remove “query” dependency in client module
2022-09-15 10:07:58 +08:00
discord9
20dcaa6897 feat: interval& None value for prev&`next (#252)
* test: for builtin functions

* test: expect fail for `datetime()`

* feat: add `interval()` fn(WIP)

* feat: `interval()` fn in builtin(UNTEST)

* refactor: move `py_vec_obj_to_array` to util.rs

* style: fmt

* test: simple `interval()` cases

* test: `interval()` with `last()`&`first()`

* doc: `ts` param of `interval()`

* log: common_telemetry for logging in script crate

* doc: corrsponding test fn for each .ron file

* feat: change to`mpsc` for schedule_job

* test: schedule_job

* dep: rm rustpython dep in common-function

* refactor: mv `schedule_job` into `Script` trait

* test: change to use `interval` to sample datapoint

* feat: add gen_none_array for generate None Array

* feat: impl Missing value for `prev`&`next`

* test: `sum(prev(values))`

* doc: add comment for why not support Float16 in `prev()`

* feat: add `interval` in py side mock module

* style: cargo fmt

* refactor: according to comments

* refactor: extract `apply_interval_function`

* style: cargo fmt

* refactor: remove `schedule()`

* style: cargo fmt
2022-09-14 10:48:27 +08:00
LFC
ec99eb0cd0 feat: frontend instance (#238)
* feat: frontend instance

* no need to carry column length in `Column` proto

* add more tests

* rebase develop

* create a new variant with already provisioned RecordBatches in Output

* resolve code review comments

* new frontend instance does not connect datanode grpc

* add more tests

* add more tests

* rebase develop

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 17:10:22 +08:00
dennis zhuang
03169c4a04 feat: impl scripts table and /run-script restful api (#230)
* feat: impl scripts table and /execute restful api

* fix: test failures

* fix: test failures

* feat: impl /run_script API

* refactor: rename run_script api to run-script and test script manager

* fix: remove println

* refactor: error mod

* refactor: by CR comments

* feat: rebase develop and change timestamp/gmt_crated/gmt_modified type to timestamp

* refactor: use assert_eq instread of assert

* doc: fix comment in Script#execute function
2022-09-13 15:09:00 +08:00
Lei, Huang
9366e77407 feat: impl timestamp type, value and vectors (#226)
* wip: impl timestamp data type

* add timestamp vectors

* adapt to recent changes to vector module

* fix all unit test

* rebase develop

* fix slice

* change default time unit to millisecond

* add more tests

* fix some CR comments

* fix some CR comments

* fix clippy

* fix some cr comments

* fix some CR comments

* fix some CR comments

* remove time unit in LogicalTypeId::Timestamp
2022-09-09 11:43:30 +08:00
evenyag
7f8195861e feat: Adds push_value_ref and extend_slice_of to MutableVector (#215)
* feat: Impl cmp_element() for Vector

* chore: Add doc comments to MutableVector

* feat: Add create_mutable() to DataType

Add `create_mutable()` to create a MutableVector for each DataType.
Implement ListVectorBuilder and NullVectorBuilder for ListType and
NullType.

* feat: Add ValueRef

ValueRef is a reference to value, could be used to avoid some allocation
when getting data from Vector. To support ValueRef, also implement a
ListValueRef for ListValue, but comparision of ListValueRef still
requires some allocation, due to the complexity of ListValue and
ListVector.

Impl some From trait for ValueRef

* feat: Implement get_ref for Vector

* feat: Remove cmp_element from Vector

`cmp_element` could be replaced by `get_ref` and then compare

* feat: Implement push/extend for PrimitiveVectorBuilder

Implement push_value_ref() and extend_slice_of() for
PrimitiveVectorBuilder.

Also refactor the DataTypeBuilder trait for
primitive types to PrimitiveElement trait, adds necessary cast helper
methods to it.
- Cast a reference to Vector to reference arrow's primitive array
- Cast a ValueRef to primitive type
- Also make PrimitiveElement super trait of Primitive

* feat: Implement push/extend for all vector builders

Implement push_value_ref() and extend_slice_of() for remaining vector
builders. Add some helpful cast method to ValueRef and a method to
cast Value to ValueRef.

Change the behavior of PrimitiveElement::cast_xxx to panic when unable
to cast, since push_value_ref() and extend_slice_of() always panic
when given invalid input data type.

* feat: MutableVector returns error if data type unmatch

* test: Add tests for ValueRef

* feat: Add tests for Vector::get_ref

* feat: NullVector returns error if data type unmatch

* test: Add tests for vector builders

* fix: Fix compile error in python coprocessor

* refactor: Add lifetime param to IntoValueRef

The Primitive trait just use the `IntoValueRef<'static>` bound. Also
rename create_mutable to create_mutable_vector.

* chore: Address CR comments

* feat: Customize PartialOrd/Ord for Value/ValueRef

Panics if values/refs have different data type

* style: Fix clippy

* refactor: Use macro to generate body of ValueRef::as_xxx
2022-09-06 13:44:48 +08:00
evenyag
97d2aa4bfd feat: script engine and python impl (#219)
* feat: improve try_into_vector function

* Impl python mod and PyVector to execute script

* add AsSeq(BUT not IMPL)

* add&test pythonic_index, add into_py_obj(UNTEST)

* add into_datatypes_value(UNTEST)

* inplace setitem_by_index unsupport

* still struggle with testing AsSeq

* actually pyimpl AsSeq&AsMap

* add slice for PyVector

* improve visualibility for testing

* adjust for clippy

* add assert for test_execute_script

* add type anno in test

* feat: basic support for PyVector's operator with scalar (#64)

* feat: memory size of vector (#53)

* feat: improve try_into_vector function

* feat: impl memory_size function for vectors

* fix: forgot memory_size assertion in null vector test

* feat: use LargeUtf8 instead of utf8 for string, and rename LargeBianryArray to BinaryArray

* feat: memory_size only calculates heap size

* feat: impl bytes_allocated for memtable (#55)

* add init and constr

* rename type cast and add test

* fix bug in pyobj_to_val

* add default cast when no type specifed

* add basic add/sub/mul for array and scalar(value)

* cargo clippy

* comment out some println

* stricter clippy

* style: cargo fmt

* fix: string&bool support in val2pyobj & back

* style: remove println in test

* style: rm println in test mod in python.rs

* refactor: use wrap_index instead of pythonic_index

* refactor: right op in scalar_arith_op

* fix: stronger type& better test

* style: remove println

* fix: scalar sign/unsigned cast

* feat: improve try_into_vector function

* Impl python mod and PyVector to execute script

* add AsSeq(BUT not IMPL)

* add&test pythonic_index, add into_py_obj(UNTEST)

* add into_datatypes_value(UNTEST)

* inplace setitem_by_index unsupport

* still struggle with testing AsSeq

* actually pyimpl AsSeq&AsMap

* add slice for PyVector

* improve visualibility for testing

* adjust for clippy

* add assert for test_execute_script

* add type anno in test

* add init and constr

* rename type cast and add test

* fix bug in pyobj_to_val

* add default cast when no type specifed

* add basic add/sub/mul for array and scalar(value)

* cargo clippy

* comment out some println

* stricter clippy

* style: cargo fmt

* fix: string&bool support in val2pyobj & back

* style: remove println in test

* style: rm println in test mod in python.rs

* refactor: use wrap_index instead of pythonic_index

* refactor: right op in scalar_arith_op

* fix: stronger type& better test

* style: remove println

* fix: scalar sign/unsigned cast

* style: remove instead of comment out

* style: remove more comment out

* feat: support scalar div vector

* style: cargo fmt

* style: typo

* refactor: rename to correct var name

* refactor: directly use arrow2::array

* refactor: mv rsub&rdiv's op into a function

* test: add python expr test

* test: add test for PyList

* refactor: tweak order of arithmetics in rtruediv

* style: remove some `use`

* refactor: move `is_instance` to mod

* refactor: move fn to mod& move `use` to head

* style: cargo fmt

* fix: correct signed/unsigned cast

* refactor: wrap err msg in another fn

* style: cargo fmt

* style: remove ok_or_else for readability

* feat: add coprocessor fn(not yet impl)

* refactor: change back to wrapped_at

* fix: update Cargo.lock

* fix: update rustc version

* Update Rust Toolchain to nightly-2022-07-14

* feat: derive Eq when possible

* style: use `from` to avoid `needless_borrow` lint

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* feat: python coprocessor with type annotation (#96)

* feat: add coprocessor fn

Signed-off-by: discord9 <zglzy29yzdk@gmail.com>

* feat: cast args into PyVector

* feat: uncomplete coprocessor

* feat: erase decorator in python ast

* feat: strip decorator in ast

* fix: change parse to `Interactive`

* style: format Cargo.toml

* feat: make coprocessor actually work

* feat: move coprocessor fn out of test mod

* feat: add error handling

* style: add some comment

* feat: rm type annotation

* feat: add type annotation support

* style: move compile method to vm closure

* feat: annotation for nullable

* feat: type coercion cast in annotation

* feat: actually cast(NOT TESTED)

* fix: allow single into(type)

* refactor: extract parse_type from parser

* style: cargo fmt

* feat: change to Expr to preserve location info

* feat: add CoprParse to deal parse check error

* style: add type anno doc for coprocessor

* test: add some test

* feat: add underscore as any type in annotation

* test: add parse& runtime testcases

* style: rm dbg! remnant

* style: cargo fmt

* feat: add more error prompt info

* style: cargo fmt

* style: add doc tests' missing `use`

* fix: doc test for coprocessor

* style: cargo fmt

* fix: add missing `use` for `cargo test --doc`

* refactor: according to reviews

* refactor: more tweaks according to reviews

* refactor: merge match arm

* refactor: move into different files(UNCOMPLELTE)

* refactor: split parse_copr into more function

* refactor: split `exec_coprocessor` to more fn

* style: cargo fmt

* feat: print Py Exceptions in String

* feat: error handling conform standards

* test: fix test_coprocessor

* feat: remove `into` in python

* test: remove all `into` in python test

* style: update comment

* refactor: move strip compile fn to impl Copr

* refactor: move `gen_schema` to impl copr

* refactor: move `check_cast_type` to impl copr

* refactor: if let to match

* style: cargo fmt

* refactor: better parse of keyword arg list

* style: cargo fmt

* refactor: some error handling(UNCOMPLETE)

* refactor: error handling to general Error type

* refactor: rm some Vec::new()

* test: modify all tests to ok

* style: reorder item

* refactor: fetch using iter

* style: cargo fmt

* style: fmt macro by hand

* refactor: rename InnerError to Error

* test: use ron to write test

* test: add test for exec_copr

* refactor: add parse_bin_op

* feat: add check_anno

* refactor: add some checker function

* refactor: exec_copr into smaller func

* style: add some comment

* refactor: add check for bin_op

* refactor: rm useless Result

* style: add pretty print for error with location

* feat: more info for pretty print

* refactor: mv pretty print to error.rs

* refactor: rm execute_script

* feat: add pretty print

* feat: add constant column support

* test: add test for constant column

* feat: add pretty print exec fn

* style: cargo fmt

* feat: add macro to chain call `.fail()`

* style: update doc for constant columns

* style: add lint to allow print in test fn

* style: cargo fmt

* docs: update some comment

* fix: ignore doctest for now

* refactor: check_bin_op

* refactor: parse_in_op, check ret anno fn

* refactor: rm check_decorator

* doc: loc add newline explain

* style: cargo fmt

* refactor: use Helper::try_into_vec in try_into_vec

* style: cargo fmt

* test: add ret anno test

* style: cargo fmt

* test: add name for .ron tests for better debug

* test: print emoji in test

* style: rm some comment out line

* style: rename `into` to `try_into` fn

* style: cargo fmt

* refactor: rm unuse serialize derive

* fix: pretty print out of bound fix

* fix: rm some space in pretty print

* style: cargo fmt

* test: not even a python fn def

* style: cargo fmt

* fix: pretty print off by one space

* fix: allow `eprint` in clippy lint

* fix: compile error after rebase develop

* feat: port 35 functions from DataFusion to Python Coprocessor (#137)

* refactor: `cargo clippy`

* feat: create a module

* style: cargo fmt

* feat: bind `pow()` function(UNTEST)

* test: add test for udf mod

* style: allow part eq not eq for gen code

* style: allow print in test lint

* feat: use PyObjectRef to handle more types

* feat: add cargo feature for udf modules

* style: rename feature to udf-builtins

* refactor: move away from mod.rs

* feat: add all_to_f64 cast fn

* feat: add bind_math_fn macro

* feat: add all simple math UDF

* feat: add `random(len)` math fn

* feat: port `avg()` from datafusion

* refactor: add `eval_aggr_fn`

* feat: add bind_aggr_fn macro

* doc: add comment for args of macro

* feat: add all UDAF from datafusion

* refactor: extract test to separate file

* style: cargo fmt

* test: add incomplete test

* test: add .ron test fn

* feat: support scalar::list

* doc: add comments

* style: rename VagueFloat/Int to LenFloat/IntVec

* test: for all fn(expect approx_median)

* test: better print

* doc: add comment for FloatWithError

* refactor: move test.rs out of builtins/

* style: cargo fmt

* doc: add comment for .ron file

* doc: update some comments

* test: EPS=1e-12 for float eq

* test: use f64::EPSILON instead

* test: change to 2*EPS

* test: cache interpreter for fast testing

* doc: remove a TODO which is done

* test: refacto to_py_obj fn

* fix: pow fn

* doc: add a TODO for type_.rs

* test: use new_int/float in test serde

* test: for str case

* style: cargo fmt

* feat: cast PyList to ScalarValue::List

* test: cast scalar to py obj and back

* feat: cast to PyList

* test: cast from PyList

* test: nested PyVector unsupported

* doc: remove unrunable doctest

* test: replace PartialEq with impl just_as_expect

* doc: add name for discord9's TODO

* refactor: cahnge to vm.ctx.new_** instead

* doc: complete a TODO

* refactor: is_instance and other minor problem

* refactor: remove type_::is_instance

* style: cargo fmt

* feat: rename to `greptime_builtin`

* fix: error handling for PyList datatype

* style: fix clippy warning

* test: for PyList

* feat: Python Coprocessor MVP (#180)

* feat: add get_arrow_op

* feat: add comparsion op(UNTESTED)

* doc: explain why no rich compare

* refactor: py_str2str&parse_keywords

* feat: add DecoratorArgs

* refactor: parse_keywords ret Deco Args

* style: remove unused

* doc: add todo

* style: remove some unused fn

* doc: add comment for copr's field

* feat: add copr_engine module

* refactor: move to `script` crate

* style: clean up cargo.toml

* feat: add query engine for copr engine

* refactor: deco args into separate struct

* test: update corrsponding test

* feat: async coprocessor engine

* refactor: add `exec_parsed` fn

* feat: sync version of coprocessor(UNTEST)

* refactor: remove useless lifetime

* feat: new type for async stream record batch

* merge: from PR#137 add py builtins

* toolchain: update rustc to nightly-08-16

* feat: add `exec_with_cached_vm` fn(Can't compile)

* toolchain: revert to 07-14

* fix: `exec_with_cached_vm`

* fix: allow vector[_] in params

* style: cargo fmt

* doc: update comment on `_`&`_|None`

* fix: allow import&ignore type anno is ok

* feat: allow ignore return types

* refsctor: remove unused py files in functions/

* style: fmt&clippy

* refactor: python modules (#186)

* refactor: move common/script to script

* fix: clippy warnings and refactor python modules

* refactor: remove modules mod rename tests mod

* feat: adds Script and ScriptEngine trait, then impl PyScript/PyScriptEngine

* refactor: remove pub use some functions in script

* refactor: python error mod

* refactor: coprocessor and vector

* feat: adds engine test and greptime.vector function to create vector from iterable

* fix: adds a blank line to cargo file end

* fix: compile error after rebase develop

* feat: script endpoint for http server (#206)

* feat: impl /scripts API for http server

* feat: adds http api version

* test: add test for scripts handler and endpoint

* feat: python side mock module and more builtin functions (#209)

* feat: add python side module(for both mock and real upload script)

* style: add *.pyc to gitignore

* feat: move copr decorator(in .py) to greptime.py

* doc: update comment for `datetime`&`mock_tester`&gitignore

* feat: `filter()` a array with bool array(UNTESTED)

* feat: `prev()`ious elem in array ret as new array(UNTEST)

* feat: `datetime()` parse date time string and ret integer(UNTEST)

* fix: add missing return&fmt

* fix: allow f32 cast to PyFloat

* fix: `datetime()`'s last token now parsed

* test: `calc_rvs` now can run with builtin module

* feat: allow rich compare which ret bool array

* feat: logic and(`&`) for bool array

* style: cargo fmt

* feat: index PyVector by bool array

* feat: alias `ln` as `log` in builtin modules

* feat: logic or(`|`)&not( `~`) for bool array

* feat: add `post` for @copr in py side mod

* feat: change datetime return to i64

* feat: py side mod `post` script to given address

* fix: add `engine` field in `post` in py side mod

* refactor: use `ConstantVector` in `pow()` builtin

* fix: prev ret err for zero array

* doc: rm comment out code

* test: incomplete pyside mod test case

* git: ignore all __pycache__

* style: fmt&clippy

* refactor: split py side module into exmaple&gptime

* feat: init_table in py using `v1/sql`  api

* feat: calc_rvs now run both locally and remote

* doc: add doc for how to run it

* fix: comment out start server code in test

* fix: clippy warnings

* fix: http test url

* fix: some CR problems

* fix: some CR problems

* refactor: script executor for instance

* refactor: remove engine param in execute_script

* chore: Remove unnecessary allow attributes

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Discord9 <discord9@163.com>
Co-authored-by: discord9 <zglzy29yzdk@gmail.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
2022-09-01 20:38:39 +08:00