227 Commits

Author SHA1 Message Date
Ning Sun
a362c4fb22 chore: use explicit geo_json version 2022-11-01 16:12:01 +08:00
Ning Sun
20fcc7819a feat: implement postgres read/write for geo data 2022-11-01 16:03:33 +08:00
Ning Sun
8119bbd7b4 feat: update sqlparser api 2022-10-20 10:57:11 +08:00
Ning Sun
48693cb12d fix: rename wkt conversion function 2022-10-19 14:52:48 +08:00
Ning Sun
8d938c3ac8 feat: use our forked sqlparser to parse Geometry(POINT) syntax 2022-10-19 14:13:26 +08:00
liangxingjian
3235436f60 fix: format 2022-10-19 10:31:02 +08:00
liangxingjian
3bf2c9840d feat: add impl of arrow array access 2022-10-19 10:25:58 +08:00
liangxingjian
35afa9dc74 fix: fix some error 2022-10-18 15:09:45 +08:00
liangxingjian
f4d8c2cef6 feat: implement simple sql demo 2022-10-18 11:54:38 +08:00
Ning Sun
788001b4bc fix: resolve lint warnings 2022-10-18 11:32:42 +08:00
Ning Sun
92ab3002c9 fix: resolve check warnings 2022-10-18 11:27:59 +08:00
Ning Sun
36ce08cb03 refactor: set inner subtype to ConcreteDataType::Geometry 2022-10-18 11:16:14 +08:00
liangxingjian
2f159dbe22 feat:add impl of geo-vec,add some unit test 2022-10-14 18:04:12 +08:00
liangxingjian
3d7d029cb5 feat:add some impl and test with a little refactor 2022-10-13 18:12:19 +08:00
liangxingjian
7aed777bc4 feat:add iter and ref of geo types 2022-10-12 17:06:42 +08:00
liangxingjian
ebcd18d3c4 feat:add some impl of geo type 2022-10-11 19:07:15 +08:00
liangxingjian
d44887bada feat:add some geo vec impl 2022-10-10 17:52:17 +08:00
liangxingjian
c0893ac19b fix:fix some error 2022-10-10 15:23:38 +08:00
liangxingjian
8a91e26020 feat:init to add new geo types 2022-10-09 17:46:06 +08:00
Ning Sun
178f8b64b5 fix: Update pgwire and fix buffer overflow issue (#293) 2022-09-29 17:58:03 +08:00
fys
fe8327fc78 feat: support write data via influxdb line protocol in frontend (#280)
* feat: support influxdb line protocol write
2022-09-29 17:08:08 +08:00
evenyag
ed89cc3e21 feat: Change signature of the Region::alter method (#287)
* feat: Change signature of the Region::alter method

* refactor: Add builders for ColumnsMetadata and ColumnFamiliesMetadata

* feat: Support altering the region metadata

Altering the region metadata is done in a copy-write fashion:
1. Convert the `RegionMetadata` into `RegionDescriptor` which is more
   convenient to mutate
2. Apply the `AlterOperation` to the `RegionDescriptor`. This would
   mutate the descriptor in-place
3. Create a `RegionMetadataBuilder` from the descriptor, bump the
   version and then build the new metadata

* feat: Implement altering table using the new Region::alter api

* refactor: Replaced wal name by region id

Region id is cheaper to clone than name

* chore: Remove pub(crate) of build_xxxx in engine mod

* style: fix clippy

* test: Add tests for AlterOperation and RegionMetadata::alter

* chore: ColumnsMetadataBuilder methods return &mut Self
2022-09-28 13:56:25 +08:00
Lei, Huang
25078e821b feat: type rewrite optimizer (#272)
* feat: add type conversion optimizer

* feat: add expr rewrite logical plan optimizer

* chore: add some doc

* fix: unit test

* fix: time zone issue in unit tests

* chore: add more tests

* fix: some CR comments

* chore: rebase develop

* chore: fix unit tests

* fix: unit test use timestamp with time zone

* chore: add more tests
2022-09-28 13:56:13 +08:00
LFC
ca732d45f9 feat: opentsdb support (#274)
* feat: opentsdb support

* fix: tests

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* fix: resolve CR comments

* refactor: remove feature flags for opentsdb and pg

* fix: resolve CR comments

* fix: resolve CR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-26 15:47:43 +08:00
dennis zhuang
0fa68ab7a5 feat: show databases and show tables (#276)
* feat: ensure time index column can't be included in primary key

* feat: sql parser supports show tables statement

* feat: impl show databases and show tables, #183

* feat: impl like expression for show databases/tables and add tests

* fix: typo

* fix: address CR problems
2022-09-26 14:05:49 +08:00
dennis zhuang
5f322ba16e feat: impl default constraint for column (#273)
* feat: impl default value for column in schema

* test: add test for column's default value

* refactor: rename ColumnDefaultValue to ColumnDefaultConstraint

* fix: timestamp column may be a constant vector

* fix: test_shutdown_pg_server

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: typo

Co-authored-by: LFC <bayinamine@gmail.com>

* chore: use table_info directly

Co-authored-by: LFC <bayinamine@gmail.com>

* refactor: by CR comments

Co-authored-by: LFC <bayinamine@gmail.com>
2022-09-22 10:43:21 +08:00
evenyag
a954ba862a feat: Implement dedup reader (#270)
* feat: Handle empty NullVector in replicate_null

* chore: Rename ChunkReaderImpl::sst_reader to batch_reader

* feat: dedup reader wip

* feat: Add BatchOp

Add BatchOp to support dedup/filter Batch and implement BatchOp for
ProjectedSchema.

Moves compare_row_of_batch to BatchOp::compare_row.

* feat: Allow Batch has empty columns

* feat: Implement DedupReader

Also add From<MutableBitmap> for BooleanVector

* test: Test dedup reader

Fix issue that compare_row compare by full key not row key

* chore: Add comments to BatchOp

* feat: Dedup results from merge reader

* test: Test merge read after flush

* test: Test merge read after flush and reopen

* test: Test replicate empty NullVector

* test: Add tests for `ProjectedSchema::dedup/filter`

* feat: Filter empty batch in DedepReader

Also fix clippy warnings and refactor some codes
2022-09-21 17:49:53 +08:00
evenyag
9489862417 fix: Fix sequence decrease after flush then reopen (#271)
The log store use start sequence instead of file start id to filter
log stream. Add more tests about flush, including flush empty memtable
and reopen after flush
2022-09-21 14:23:59 +08:00
Lei, Huang
35ba0868b5 feat: impl filter push down to parquet reader (#262)
* wip add predicate definition

* fix value move

* implement predicate and prune

* impl filter push down in chunk reader

* add more expr tests

* chore: rebase develop

* fix: unit test

* fix: field name/index lookup when building pruning stats

* chore: add some meaningless test

* fix: remove unnecessary extern crate

* fix: use datatypes::schema::SchemaRef
2022-09-21 11:47:55 +08:00
Ning Sun
8a400669aa feat: postgre wire protocol for frontend (#269) 2022-09-19 15:39:53 +08:00
evenyag
e697ba975b feat: Implement dedup and filter for vectors (#245)
* feat: Dedup vector

* refactor: Re-export Date/DateTime/Timestamp

* refactor: Named field for ListValueRef::Ref

Use field val instead of tuple for variant ListValueRef::Ref to keep
consistence with ListValueRef::Indexed

* feat: Implement ScalarVector for ListVector

Also implements ScalarVectorBuilder for ListVectorBuilder, Scalar for
ListValue and ScalarRef for ListValueRef

* test: Add tests for ScalarVector implementation of ListVector

* feat: Implement dedup using match_scalar_vector

* refactor: Move dedup func to individual mod

* chore: Update ListValueRef comments

* refactor: Move replicate to VectorOp

Move compute operations to VectorOp trait and acts as an super trait of
Vector. So we could later put dedup/filter methods to VectorOp trait,
avoid to define too many methods in Vector trait.

* refactor: Move scalar bounds to PrimitiveElement

Move Scalar and ScalarRef trait bounds to PrimitiveElement, so for each
native type which implements PrimitiveElement, its PrimitiveVector
always implements ScalarVector, so we could use it as ScalarVector
without adding additional trait bounds

* refactor: Move dedup to VectorOp

Remove compute mod and move dedup logic to operations::dedup

* feat: Implement VectorOp::filter

* test: Move replicate test of primitive to replicate.rs

* test: Add more replicate tests

* test: Add tests for dedup and filter

Also fix NullVector::dedup and ConstantVector::dedup

* style: fix clippy

* chore: Remove unused scalar.rs

* test: Add more tests for VectorOp and fix failed tests

Also fix TimestampVector eq not implemented.

* chore: Address CR comments

* chore: mention vector should be sorted in comment

* refactor: slice the vector directly in replicate_primitive_with_type
2022-09-19 14:05:02 +08:00
LFC
a649f34832 fix: select empty table (#268)
* fix: select empty table

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-19 11:28:12 +08:00
Lei, Huang
1770079691 fix: slice implementation for DateVector/DateTimeVector… (#266)
* fix: replicate and slice implementation for DateVector/DateTimeVector/TimestampVector

* chore: rebase develop
2022-09-16 16:38:46 +08:00
Ning Sun
1639b6e7ce refactor: rename to_vec to take for RecordBatches (#264) 2022-09-16 14:04:04 +08:00
Ning Sun
e67b0eb259 feat: Initial support of postgresql wire protocol (#229)
* feat: initial commit of postgres protocol adapter

* initial commit of postgres server

* feat: use common_io runtime and correct testcase

* fix previous tests

* feat: adopt pgwire api changes and add support for text encoded data

* feat: initial integration with datanode

* test: add feature flag to test

* fix: resolve lint warnings

* feat: add postgres feature flags for datanode

* feat: add support for newly introduced timestamp type

* feat: adopt latest datanode changes

* fix: address clippy warning for flattern scenario

* fix: make clippy great again

* fix: address issues found in review

* chore: sort dependencies by name

* feat: adopt new Output api

* fix: return error on unsupported data types

* refactor: extract common code dealing with record batches

* fix: resolve clippy warnings

* test: adds some unit tests postgres handler

* test: correct test for cargo update

* fix: update query module name

* test: add assertion for error content
2022-09-15 21:39:05 +08:00
LFC
fb6153f7e0 feat: a new type for supplying Ord to Primitive (#255)
Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-15 18:32:55 +08:00
dennis zhuang
dfa3012396 feat: improve python coprocesssor parser (#260)
* feat: supports DateTime, Date and Timestamp column type to be returned by py scripts

* feat: improve coprocessor compiler, make it work better

* fix: comments

* fix: typo

* Update src/script/src/python/vector.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/script/src/python/coprocessor.rs

Co-authored-by: LFC <bayinamine@gmail.com>

Co-authored-by: LFC <bayinamine@gmail.com>
2022-09-15 16:18:33 +08:00
dennis zhuang
c8cb705d9e ci: pre-commit configuration and hooks (#261)
* feat: adds pre-commit config and hooks

* refactor: sort all Cargo.toml by cargo-sort

* ci: adds conventional-pre-commit hook to pre-commit

* fix: remove .pre-commit-hooks.yaml

* fix: readme

* Update .pre-commit-config.yaml

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>

* ci: move clippy hook to push stage

* docs: install pre-push github hook

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
2022-09-15 11:30:08 +08:00
fys
8400f8dfd4 chore: move query::Output to common-query module (#259)
* chore: move query::Output to common-query module

* chore: remove “query” dependency in client module
2022-09-15 10:07:58 +08:00
fys
ef40b12749 chore: add optional for datatype, rename data_type to datatype (#258) 2022-09-14 18:07:22 +08:00
evenyag
0dce8946d4 ci: Add ci cache (#256) 2022-09-14 16:06:59 +08:00
LFC
7dee7199dc fix: set unittests dir to /tmp can explode grcov's disk (#253)
* fix: set unittests dir to /tmp can explode grcov's disk

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-14 15:10:10 +08:00
Lei, Huang
2dbaad9770 fix: forbid use int64 as timestamp column data type (#248)
* fix: forbid use int64 as timestamp column data type

* fix unit test

* fix unit tests

* change gmt_created and gmt_modified data type in system tables to timestamp

* also change data type in readme
2022-09-14 12:03:16 +08:00
discord9
20dcaa6897 feat: interval& None value for prev&`next (#252)
* test: for builtin functions

* test: expect fail for `datetime()`

* feat: add `interval()` fn(WIP)

* feat: `interval()` fn in builtin(UNTEST)

* refactor: move `py_vec_obj_to_array` to util.rs

* style: fmt

* test: simple `interval()` cases

* test: `interval()` with `last()`&`first()`

* doc: `ts` param of `interval()`

* log: common_telemetry for logging in script crate

* doc: corrsponding test fn for each .ron file

* feat: change to`mpsc` for schedule_job

* test: schedule_job

* dep: rm rustpython dep in common-function

* refactor: mv `schedule_job` into `Script` trait

* test: change to use `interval` to sample datapoint

* feat: add gen_none_array for generate None Array

* feat: impl Missing value for `prev`&`next`

* test: `sum(prev(values))`

* doc: add comment for why not support Float16 in `prev()`

* feat: add `interval` in py side mock module

* style: cargo fmt

* refactor: according to comments

* refactor: extract `apply_interval_function`

* style: cargo fmt

* refactor: remove `schedule()`

* style: cargo fmt
2022-09-14 10:48:27 +08:00
LFC
ec99eb0cd0 feat: frontend instance (#238)
* feat: frontend instance

* no need to carry column length in `Column` proto

* add more tests

* rebase develop

* create a new variant with already provisioned RecordBatches in Output

* resolve code review comments

* new frontend instance does not connect datanode grpc

* add more tests

* add more tests

* rebase develop

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 17:10:22 +08:00
LFC
bdd5bdd917 Set unittest's logging dir to "/tmp" to not pollute source codes' dir when running unittests from IDE. (#249)
Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 16:14:32 +08:00
dennis zhuang
03169c4a04 feat: impl scripts table and /run-script restful api (#230)
* feat: impl scripts table and /execute restful api

* fix: test failures

* fix: test failures

* feat: impl /run_script API

* refactor: rename run_script api to run-script and test script manager

* fix: remove println

* refactor: error mod

* refactor: by CR comments

* feat: rebase develop and change timestamp/gmt_crated/gmt_modified type to timestamp

* refactor: use assert_eq instread of assert

* doc: fix comment in Script#execute function
2022-09-13 15:09:00 +08:00
LFC
cad35fe82e ci: make grcov happy (#246)
* make grcov happy

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 14:19:49 +08:00
LFC
64b6b2afe1 feat: procedure macro to help writing UDAF (#242)
* feat: procedure macro to help writing UDAF

* resolve code review comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-09-13 10:39:44 +08:00
Morranto
628cdb89e8 feat: Add grpc implementation for alter table opeartions (#239)
* feat: grpc-alter impl

* fix: format

* fix cr

* Update src/datanode/src/error.rs

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

* Update src/datanode/src/server/grpc/ddl.rs

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

* fix bug

* Update src/datanode/src/server/grpc/ddl.rs

Co-authored-by: Ning Sun <sunng@protonmail.com>

* fix:format

* fix bug

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2022-09-10 21:50:21 +08:00