rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-17 02:32:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
747 Commits 106 Branches 227 Tags
d1ba9ca1261ebce264b8bc6bb26f04fa2c5ce407
Commit Graph

197 Commits

Author SHA1 Message Date
Lei, HUANG
757b4a87a0 fix: remove profile in mem-prof crate to suppress compiler warnings (#1146) 2023-03-09 03:39:24 +00:00
Eugene Tolbakov
b31a6cb506 refactor: replace tempdir with tempfile (#1123) 
* refactor: replace tempdir with tempfile

* refactor(query): move tempfile dependency under the workspace's Cargo.toml

* refactor(tempfile): create common-test-util

* refactor(tempfile): fix toml format

* refactor(tempfile): remove tempfile out of dependencies

* refactor(tempfile): fix incorrect toml
 2023-03-08 11:15:56 +08:00
Ruihang Xia
3a527c0fd5 feat: impl proc macro range_fn and some aggr_over_time functions (#1072) 
* impl range_fn proc macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl some aggr_over_time fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl present_over_time and absent_over_time

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* accomplish planner, and correct type cast

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* document the macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update irate/idelta test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-03-07 23:39:45 +08:00
Lei, HUANG
a4c01f4a3a feat: memory profiling (#1124) 
* feat: use jemalloc as default allocator

* feat: add feature for mem-prof

* feat: add errors

* make common-mem-prof optional dep

* fix: toml format

* doc: add profile doc

* fix: typo
 2023-03-07 17:12:51 +08:00
discord9
379f581780 test: add Integrated Test for Coprocessor& fix minor bugs (#1122) 
* feat: cache `Runtime`

* fix: coprstream schema not set

* test: integrated tests for Coprocessor

* fix: UDF fixed

* style: remove unused import

* chore: remove more unused import

* feat: `filter`, (r)floordiv for Vector

* chore: CR advices

* feat: auto convert to `lit`

* chore: fix typo

* feat: from&to `pyarrow.array`

* feat: allow `pyarrow.array` as args to builtins

* chore: cargo fmt

* test: CI add `pyarrow`

* test: install Python&PyArrow in CI

* test: not cache depend for now

* chore: CR advices

* test: fix name

* style: rename
 2023-03-06 19:20:59 +08:00
yuanbohan
9a87f5edf8 fix(grpc): support timestamp precision (#1113) 2023-03-02 17:33:59 +08:00
Yingwen
3fd9c2f144 feat: Store error in procedure state (#1062) 
* docs: Change comment position

* refactor(procedure): Store error in ProcedureState

* test: Mock instance with procedure enabled

* feat: Add wait method to wait for procedure

* test(datanode): Test create table by procedure

* chore: Fix clippy
 2023-03-01 14:37:50 +08:00
Ning Sun
d402f83442 ci: generate apidocs when pushing to default branch (#1093) 
* ci: generate apidocs when pushing to default branch

* ci: require clippy before running tests

* fix: resolve new clippy warnings on primitive slice

* fix: resolve more clippy warnings

* Update .github/workflows/apidoc.yml

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* ci: add an index html to redirect

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2023-03-01 13:18:26 +08:00
LFC
11d45e2918 refactor: upgrade DataFusion, Arrow and Sqlparser (#1074) 
* refactor: upgrade DataFusion, Arrow and Sqlparser

* fix: resolve PR comments
 2023-02-27 22:20:08 +08:00
Xieqijun
0b3f955ca7 feat: Add an error variant RetryLater (#1058) 
* feat: support retry error

* fix: ci

* fix: ci

* fix: fmt

* feat: add convert procedure error

* Docs : add rustdoc

* fix: cr

* fix: cr

* fix: rm unless code
 2023-02-27 17:19:37 +08:00
Ning Sun
4b58a8a18d feat: update substrait and prost version (#1080) 2023-02-27 15:18:12 +08:00
Yingwen
bd377ef329 feat: Procedure to create table and register table to catalog (#1040) 
* feat: Add table-procedures crate

* feat: Implement procedure to create table

* feat: Integrate procedure manager to datanode

* test: Test CreateTableProcedure

* refactor: Rename table-procedures to table-procedure

* feat: Implement create_table_by_procedure

* chore: Remove comment

* chore: Add todo

* feat: Add procedure config to standalone mode

* feat: Register table-procedure loaders

* feat: Address review comments

CreateTableProcedure just return error if the subprocedure is failed

* chore: Address CR comments
 2023-02-27 11:49:23 +08:00
Ruihang Xia
4cc3ac37d5 feat: add DictionaryVector DataType (#1061) 
* fix stddev and stdvar. try build range function expr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: add dictionary data type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* preserve timestamp column in range manipulator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* plan range functions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-02-23 20:31:07 +08:00
Xuanwo
98ef74bff4 chore: Bump OpenDAL to v0.27 (#1057) 
* Bump OpenDAL to v0.27

Signed-off-by: Xuanwo <github@xuanwo.io>

* Make cargo check happy

Signed-off-by: Xuanwo <github@xuanwo.io>

* Address comments

Signed-off-by: Xuanwo <github@xuanwo.io>

* Address comments

Signed-off-by: Xuanwo <github@xuanwo.io>

* Format toml

Signed-off-by: Xuanwo <github@xuanwo.io>

* Make taplo happy

Signed-off-by: Xuanwo <github@xuanwo.io>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
 2023-02-23 11:20:45 +08:00
Lei, HUANG
2df8143ad5 feat: support table ttl (#1052) 
* feat: purge expired sst on compaction

* chore: add more log

* fix: clippy

* fix: mark expired ssts as compacting before picking candidates

* fix: some CR comments

* fix: remove useless result

* fix: cr comments
 2023-02-22 16:56:20 +08:00
Yingwen
c6f2db8ae0 feat(procedure): Add procedure watcher (#1043) 
* refactor: Use watch channel to store ProcedureState

* feat: Add a watcher to wait for state change

* test: test watcher on procedure failure

* feat: Only clear message cache on success

* feat: submit returns Watcher
 2023-02-21 17:19:39 +08:00
Lei, HUANG
e17d5a1c41 feat: support table options (#1044) 
* feat: change table options from string map to a struct, add ttl and write_buffer_size

* fix: also pass table options to table meta

* feat: pass table options when opening/creating regions

* fix: CR comments
 2023-02-21 08:10:23 +00:00
Yingwen
aaaf24143d feat: Procedure to create a mito engine (#1035) 
* feat: wip

* feat: Implement procedure to create mito table

* feat: Add create_table_procedure to TableEngine

* feat: Impl dump and lock for CreateMitoTable

* feat: Impl CreateMitoTable::execute and register it to manager

* feat(common-procedure): pub local mod

* feat: Add simple test for MitoCreateTable

* style: Fix clippy

* refactor: Move create_table_procedure to a new trait TableEngineProcedure
 2023-02-21 09:40:56 +08:00
Eugene Tolbakov
7787cfdd42 refactor(datatypes): enhance MutableVector methods (#987) 
* refactor(datatypes): enhance MutableVector methods

* refactor(datatypes): address code review issues

* refactor(datatypes): address more code review issues

* refactor(datatypes): fix merge conflicts

* refactor(datatypes): address code review issues

* refactor(datatypes): address more code review issues

* refactor(datatypes): update sql delete with the newly introduced method
 2023-02-17 16:16:23 +08:00
Yingwen
ddbc97befb refactor: changes CreateTableRequest::schema to RawSchema (#1018) 
* refactor: changes CreateTableRequest::schema to RawSchema

* refactor(grpc-expr): create_table_schema returns RawSchema
 2023-02-16 16:04:17 +08:00
Ruihang Xia
a8c2b35ec6 chore: bump rust to nightly-2023-02-14 (#1019) 
* chore: bump rust to nightly-2023-02-14

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* bump statrs to 0.16

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-02-16 13:17:57 +08:00
Yingwen
04afee216e feat(procedure): Support multi-lock keys and querying procedure state from context (#1006) 
* feat: Add ContextProvider to Context

So procedures can query states of other procedures via the
ContextProvider and they don't need to hold a ProcedureManagerRef

* feat: Procedure supports acquring multiple lock keys

* test: Use multi-locks in test

* feat: Add keys_to_lock/unlock
 2023-02-15 18:04:19 +08:00
Ning Sun
cd0d58cb24 fix: correct date/time type format for postgresql (#1001) 
* fix: correct date/time type format for postgresql

* fix: tests for timestamp

* refactor: use Utc datetime for timestamp::to_chrono_datetime

* Update src/servers/Cargo.toml

Co-authored-by: LFC <bayinamine@gmail.com>

---------

Co-authored-by: LFC <bayinamine@gmail.com>
 2023-02-15 09:40:16 +00:00
shuiyisong
e17d564bf0 feat: add client tls option to channel manager config (#999) 
* feat: add client tls to channel manager config

* chore: move test to tests folder

* chore: fix license issue

* chore: fix cr issue
 2023-02-15 16:02:27 +08:00
Zheming Li
a19dee1dc0 feat: duplicate error logs into separate file (#995) 
Signed-off-by: Zheming Li <nkdudu@126.com>
 2023-02-15 14:27:32 +08:00
Lei, HUANG
75b8afe043 feat: compaction integration (#997) 
* feat: trigger compaction on flush

* chore: rebase develop

* feat: add config item max_file_in_level0 and remove compaction_after_flush

* fix: cr comments

* chore: add unit test to cover Timestamp::new_inclusive

* fix: workaround to fix future is not Sync

* fix: future is not sync

* fix: some cr comments
 2023-02-15 14:14:07 +08:00
Lei, HUANG
374acc8830 feat: compaction reader and writer (#972) 
* feat: compaction reader and writer

* feat: make ParquetWrite accept both memtable iterator and chunk reader

* feat: adapt ParquetWriter to accomodate ChunkReaderImpl

* chore: rebase develop

* wip: compile

* wip: task logic

* feat: version and manifest update

* fix: remove useless as_inner from Timestamp vectors

* feat: mark file compacting

* fix: unit test

* fix: clippy warnings

* fix: CR comment

* chore: according to cr comments, remove visit_levels from LevelMetas

* fix: some CR comments

* fix: add PlainTimestampRowFilter for correctness

* fix: cr comments

* fix: some typos
 2023-02-14 17:32:00 +08:00
Yingwen
0f7e5a2fb2 feat: Implement LocalManager::recover (#981) 
* feat: Implement LocalManager::recover

* feat: Impl From<ObjectStore> for ProcedureStore
 2023-02-14 14:50:43 +08:00
Yingwen
be897efd01 feat: Execute procedure in LocalManager (#953) 
* feat: Runner executes procedure

* feat: Add rollback key type to ParsedKey

* feat: Write rollback key when procedure is unable to execute

* feat: Use loaded step to re-submit subprocedure

* feat: Track subprocedures in ProcedureMeta

* feat: Clean message cache after the root procedure is done

* feat: Runner returns execution result

* fix: Fix tests

* test: Test Runner

* test: Test procedures_in_tree

* chore: Refine test and comments

* feat: Remove support of lock inheritance

A deadlock happens if a subprocedure acquires the same lock key as
its parent.

The main concern is if the subprocedure directly inherits its parent's
lock, then how should we behave when multiple subprocedures acquire
this same lock? Each procedure may assume it has unique access to the
same object but it actually shares the resource with others.

Now subprocedures need to use different keys to lock objects, which is
reasonable. For example:
- A parent procedure wants to create a table so it locks the table with
a key like `catalog.schema.table`
- Subprocedures create regions for the table so they lock the regions
with keys `catalog.schema.table.region-0 ~ catalog.schema.table.region-n`

* style: Fix clippy

* feat: insert_procedure returns false on duplicate procedure

Also rename this method to try_insert_procedure

* chore: Address CR comments
 2023-02-13 10:38:56 +08:00
Lei, HUANG
e77a7f253c feat: L0 to L1 compaction strategy (#964) 
* feat: impl simple compaction strategy

* chore: rebase to develop and fix clippy warnings

* chore: simplify time bucket strcut

* chore: some typos
 2023-02-11 21:10:24 +08:00
LFC
c6128ec0a4 refactor: use remote proto (#963) 
* refactor: use remote proto (see our new repo "GreptimeTeam/greptime-proto")

* fix: resolve PR comments
 2023-02-10 13:35:18 +08:00
shuiyisong
9989a8c192 fix: check full table name during logical plan creation (#948) 2023-02-09 17:23:28 +08:00
LFC
803b7f0633 feat: implement "drop table" in distributed mode (both in SQL and gRPC) (#944) 
* feat: implement "drop table" in distributed mode (both in SQL and gRPC)

refactor: create distributed table
some details:
- set table global value in Meta, as well as table routes value. Datanode only set table regional value
- complete instance SQL tests both in standalone and distributed mode

* fix: rebase develop

* fix: resolve PR comments
 2023-02-08 07:36:38 +00:00
Yingwen
ecadbc1435 feat: Add procedure manager LocalManager (#946) 
* feat: Add ManagerContext and LocalManager

* test: Add register_loader test

* feat: Remove some unused methods

* fix: Fix submit_procedure ensure condition
 2023-02-07 11:33:13 +08:00
Lei, HUANG
5d62e193bd feat: support multi regions on datanode (#653) 
* wip: fix compile errors

* chore: move splitter to partition crate

* fix: remove useless variants in frontend errors

* chore: move more partition related code to partition manager

* fix: license header

* wip: move WriteSplitter to PartitionRuleManager

* fix: clippy warnings

* chore: remove useless error variant and format toml

* fix: cr comments

* chore: resolve conflicts

* chore: rebase develop

* fix: cr comments

* feat: support multi regions on datanode

* chore: rebase onto develop

* chore: rebase develop

* chore: rebase develop

* wip

* fix: compile errors

* feat: multi region

* fix: CR comments

* feat: allow stat existing regions without actually open it

* fix: use table meta in manifest to recover region info
 2023-02-07 10:46:18 +08:00
Yingwen
b0925d94ed feat: Implement lock component for ProcedureManager (#937) 
* feat: Add procedure meta

* feat: Implement lock for procedures

* chore: Allow dead code

* docs: Fix comment

* docs: Update docs of acquire_lock
 2023-02-03 18:42:03 +08:00
LFC
af935671b2 feat: support "use" in GRPC requests (#922) 
* feat: support "use catalog and schema"(behave like the "use" in MySQL) in GRPC requests

* fix: rebase develop
 2023-02-02 20:02:56 +08:00
Yingwen
74adb077bc feat: Implement ProcedureStore (#927) 
* test: Add more tests for ProcedureId

* feat: Add ObjectStore based state store

* feat: Implement ProcedureStore

* test: Add tests for ParsedKey

* refactor: Rename list to walk_top_down

* fix: Test ProcedureStore and handles unordered key values.

* style: Fix clippy

* docs: Update comment

* chore: Adjust log level for printing invalid key
 2023-02-02 17:49:31 +08:00
Ruihang Xia
ea5146762a chore(deps): bump promql-parser (#929) 
* fix promql crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* migrate to new api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix aggregator test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix styles

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-02-02 07:31:41 +00:00
Yingwen
b2ad0e972b feat: Define procedure related traits (#904) 
* chore: Move uuid to workspace.dependencies

* feat: Define procedure related traits

* test: Add tests

* chore: Update imports

* feat: Submit ProcedureWithId to manager

* chore: pub ProcedureId::parse_str

* refactor: ProcedureId::parse_str returns Result

* chore: Address CR comments

Also implements FromStr for ProcedureId
 2023-01-31 14:17:28 +08:00
Lei, HUANG
5e05c8f884 fix: TimestampRange::new_inclusive and strum dependency (#910) 
fix: TimestampRange::new_inclusive; also fix strum dependency in common-error
 2023-01-29 13:09:05 +08:00
LFC
64243e3a7d refactor: accommodate java flight client (#886) 
* refactor: change how AffectedRows is carried in flight stream to accommodate Java Flight client

* fix: clippy
 2023-01-29 11:27:13 +08:00
Ruihang Xia
36a13dafb7 build(deps): bump tokio to 1.24.2 (#900) 
deps: bump tokio to 1.24.2

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-01-29 11:13:37 +08:00
Lei, HUANG
43aefc5d74 feat: prunine sst files according to time range in filters (#887) 
* 1. Reimplement Eq for Timestamp
2. Add and/or for GenericRange

* feat: extract time range from filters

* feat: select sst files according to time range

* fix: clippy

* fix: empty value in range

* fix: some cr comments

* fix: return optional timestamp range

* fix: cr comments
 2023-01-28 15:16:41 +08:00
shuiyisong
6960739b3d feat: add authorize to UserProvider trait (#879) 
* feat: add SchemaValidator

* feat: add schema validator to mysql shim

* chore: pass schema validator to http auth layer

* feat: add schema validator to http

* feat: add schema validator to pg

* feat: add schema validator to pg

* feat: add schema validator test

* chore: remove println in test

* chore: use !matches

* refactor: refac authenticate and authorize in http auth

* refactor: refac authenticate and authorize in http auth

* chore: typo

* chore: minor change

* refactor: merge schema_validator into user_providier

* chore: fix license issue

* refactor: change http query param from database to db

* chore: fix cr issue
 2023-01-18 12:42:08 +08:00
Lei, HUANG
daad38360f fix: impl total order for Timestamp (#878) 
* 1. Reimplement Eq for Timestamp
2. Add and/or for GenericRange

* chore: add test for TimestampRange with diff unit

* chore: optimize split implementation

* fix: clippy

* fix: add fast path

* fix: CR comments
 2023-01-16 17:37:30 +08:00
discord9
e428a84446 feat: use Python Script as UDF in SQL (#839) 
* feat: reg PyScript as UDF

* refactor: use `ConcreteDataType` instead

* fix: accept `str` data type

* fix: allow binary to capture SIGINT

* test: add test for py udf

* Update src/servers/tests/py_script/mod.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* style: clippy problem

* style: add newline

* chore: PR advices

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-01-13 14:35:03 +08:00
elijah
0e8411c2ff chore: add custom log level support for common_telemetry::init_default_ut_logging() (#864) 
chore: improve default ut logging
 2023-01-11 16:52:21 +08:00
Ruihang Xia
32d51947a4 refactor: adjust outermost error message (#859) 
* refactor: adjust outermost error message

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* preserve tonic status code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-01-10 17:28:27 +08:00
Lei, HUANG
90fcaa8487 feat: expose wal config (#852) 
* feat: wal config

* fix: use human-readable string in wal config

* feat: copy ReadableSize and humanize size config items in toml files

* fix: clippy
 2023-01-10 16:07:26 +08:00