Commit Graph

1236 Commits

Author SHA1 Message Date
Lei, Huang
008f62afc1 feat: buffer abstraction (#51)
* feat: add buffer abstraction and rewrite entry encode/decode process

* add some tests

* remove pad.rst

* fix some comments

* fix comments

* remove mmap mod

* feat: Bytes type implementation switch to bytes::Bytes

* fix: use Bytes::from(String) and Bytes::from(Vec<u8>)

* feat: add new method to Entry trait
2022-07-04 14:08:23 +08:00
evenyag
11bf970efd feat: Implement TimestampMillis and RangeMillis (#56) 2022-06-29 20:55:27 +08:00
dennis zhuang
379d2e2f50 feature: runtime crate and global runtimes (#49)
* feat: init common runtime crate

* feat: tokio Runtime wrapper and global runtime functions

* feat: adds block_on_read, block_on_write, block_on_bg functions to runtime

* refactor: panic when configure global runtimes which are already initialized

* refactor: adds read/write/bg thread pool size

* fix: code style

* fix: clippy warning

* fix: test_metric panic

* fix: address CR problems

* log: adds log when creating runtime
2022-06-21 16:09:15 +08:00
evenyag
6ec870625f refactor: Refactor usage of BoxedError (#48)
* feat: Define a general boxed error

* refactor: common_function use Error in common_query

* feat: Add tests to define_opaque_error macro

* refactor: Refactor table and table engine error

* refactor: recordbatch remove arrow dev-dependency

* refactor: datanode crate use common_error::BoxedError

* chore: Fix clippy

* feat: Returning source status code when using BoxedError

* test: Fix opaque error test

* test: Add tests for table::Error & table_engine::Error

* test: Add test for RecordBatch::new()

* test: Remove generated tests from define_opaque_error

* chore: Address cr comment
2022-06-21 15:24:45 +08:00
dennis zhuang
4071b0cff2 feat: impl scanning data from storage engine for table (#47)
* feat: impl scanning data from storage for MitoTable

* adds test mod to setup table engine test

* fix: comment error

* fix: boyan -> dennis in todo comments

* fix: remove necessary send in BatchIteratorPtr
2022-06-20 15:42:57 +08:00
dennis zhuang
e78c015fc0 TableEngine and SqlHandler impl (#45)
* Impl TableEngine, bridge to storage

* Impl sql handler to process insert sql

* fix: minor changes and typo

* test: add datanode test

* test: add table-engine test

* fix: code style

* refactor: split out insert mod from sql and minor changes by CR

* refactor: replace with_context with context
2022-06-17 11:36:49 +08:00
Lei, Huang
e03ac2fc2b Implement log store append and file set management (#43)
* add log store impl

* add some test

* delete failing test

* fix: concurrent close issue

* feat: use arcswap to replace unsafe AtomicPtr

* fix: use lock to protect rolling procedure.
fix: use try_recv to replace poll_recv on appender task.

* chores: 1. use direct tmp dir instead of creating TempDir instance; 2. inline some short function; 3. rename some structs; 4. optimize namespace to arc wrapper inner struct.
2022-06-16 19:09:09 +08:00
fengjiachun
725a261b55 feat(cmd): command refactor (#44)
* feat(cmd): command refactor
2022-06-15 20:08:00 +08:00
evenyag
4171173b76 feat: Support creating in memory region and writing to memtable (#40)
* chore(store-api): Fix typo in region comments

* feat(storage): Init storage crate

* feat(store-api): Make some method async

* feat(storage): Blank StorageEngine implementation

* feat(storage): StorageEngine returns owned SchemaRef

* feat: pub use arrow in datatypes

* feat(store-api): Implement RegionMetadata

* feat(storage): Impl create region in memory.

* chore(object-store): Format cargo toml

* chore(storage): Log on region created

* feat: Impl CowCell

* feat: Store id to cf meta mapping

* refactor: Refactor version and rename it to VersionControl

* feat: Impl write batch for put, refactor column family

* feat(storage): Skeleton of writing to memtable

* refactor(storage): MemTable returns MemTableSchema

* feat: Add ColumnSchema and conversion between schema and arrow's schema

* feat: Validate put data

* feat: Valid schema of write batch

* feat: insert memtable WIP

* feat: Impl Inserter for memtable

* feat(datatypes): Implement Eq/Ord for Value

feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes

test: Test Value::from()

* feat: Define BTreeMemTable

* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.

* feat: Impl BTreeMemTable::write()

* refactor: Remove useless ColumnFamilyHandle now

* chore: Clean comment

* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value

* test(storage): Add tests for WriteBatch

* chore: Fix clippy

* feat: Add builder for RowKey/ColumnFamilyDescriptor

* test: Add test for metadata

* chore: Fix clippy

* test: Add test for region and engine

* chore: Fix clippy

* chore: Address CR comment
2022-06-09 16:50:02 +08:00
dennis zhuang
f7136819fc function crate and scalars function (#39)
* feat: adds scalars mod and enhance vectors

* temp commit

* fix compile error

* Impl pow function with new udf framework

* Adds common-function crate and impl scalars function

* fix: remove used code

* test: adds test for function crate and refactor vectors

* fix: fmt style

* feat: impl numpy.clip function

* feat: improve clip function returning int64 type when arguments do not have float type

* feat: adds more test for vectors

* feat: adds replicate method test for primitive vector

* fix: by code review

* feat: clip function returns uint64 when all arguments type are unsigned

* refactor: improve vectors#only_null

* fix: clippy warning

* fix: clippy warning

* fix: clip should return float64 when arguments have both signed and unsigned types
2022-06-08 13:15:22 +08:00
dennis zhuang
a2331366f6 feat: adds adds register_udf api to query engine and refactor datatypes (#34)
* feat: adds ColumnarValue and refactor vectors

* fix: ConcreteDataType compile error

* feat:adds udf/function mods

* feat: adds test for common_query crate

* feat: adds register_udf api to query engine

* feat: adds common_query::error test

* refactor: by CR comments

* refactor: adds impl_new_concret_type_functions! macro to reduce boilerplate codes

* fix: typo
2022-05-24 16:50:56 +08:00
Lei, Huang
06b592f00f feat: add WAL definitions (#35)
* feat: add WAL definitions

* rename and add some tests
2022-05-24 16:12:23 +08:00
evenyag
1594da337f feat(store-api): Prototype of storage engine api (#33) 2022-05-20 18:51:51 +08:00
dennis zhuang
b0d2e2e91b feat: adds ConcretDataType and more datatypes impl (#31)
* feat: adds ConcretDataType and impl binary/boolean/null types and vectors

* feat: adds String to ConcretDataType

* docs:  ConcretDataType::from_arrow_type may panic
2022-05-19 11:21:11 +08:00
evenyag
5777732fde feat(store-api): Init store-api crate (#30) 2022-05-18 17:19:57 +08:00
Lei, Huang
bd4fe1f5bc feat: RecordBatch serialization (#26) 2022-05-17 17:01:00 +08:00
evenyag
3d374cce68 feat: implement log related macros (#29) 2022-05-17 16:00:17 +08:00
dennis zhuang
a819b6bce5 feat: impl object-store based on opendal (#27)
* feat: impl object-store based on opendal

* feat: adds s3 test

* feat: export Layer

* feat: pass s3 access key/secret by env vars

* fix: logging when s3 test is running
2022-05-17 15:58:27 +08:00
dennis zhuang
93cbdbee9a feat: adds common-telemetry crate and adds logs/metrics to query engine (#25)
* feat: adds common-telemetry crate and logging mod

* refactor: common telemetry

* feat: adds metric mod and exports metrics by http

* feat: adds metrics to query engine and datanodes

* refactor: rename metrics

* refactor: rename ElapsedTimer struct and elapsed_timer macro

* refactor: log error when fail to new data node.

* fix: use backtrace crate instead of unstable feature

* feat: add must_use attr to Timer

* fix: only reserve Debug attribute for Timer
2022-05-12 17:42:48 +08:00
Lei, Huang
641f4b39bd feat: add serialization support to existing data types (#24)
* feat: add serialization support to existing data types

* fix: refactor errors mod to error. fix other review comments
2022-05-11 17:17:27 +08:00
dennis zhuang
df1a99e27f feat: test http server api (#21)
* feat: adds http server test

* feat: limit sql output to 10 rows
2022-05-09 14:54:03 +08:00
evenyag
63d9aa1bff refactor: Refactor datanode error and impl ErrorExt for it 2022-05-07 11:34:23 +08:00
evenyag
7e2e3e3429 feat: Impl ErrorExt for opaque error and ParseError 2022-05-06 20:04:16 +08:00
evenyag
aa1a6b2205 feat: Add common-error crate and implement opaque error type. 2022-05-06 20:04:14 +08:00
dennis zhuang
f0db878bf5 feat: http server and cmd crate etc. (#15)
* feat: adds cmd crate and http server

* feat: impl sql http handler

* feat: convert all arrow array types

* feat: adds query test

* feat: adds test for datanode

* fix: format

* feat: refactor state.rs

* feat: adds collect test

* fix: by code review

* fix: style
2022-05-06 17:34:29 +08:00
Lei, Huang
f889ed5488 feat: INSERT statement and planner implementation (#16)
* feat: Add SQL parser definition and SHOW DATABASE implementation

* chores: Eliminate clippy warnings and errors.

* chores: remove Gt prefix in some structs; rename some mod; remove print(s) in unit tests; refactor crate layout;
feat: wrap sqlparser error;

* chores: reorder cargo dependency

* chores: fix code style

* chores: add #[cfg(test)] to unit tests

* style: fix test mod style

* feat: implement select query parser

* chores: remove unused dependency

* feat: implement TryInto<sqlparser::ast::Statement> for Statement

* chore: fix style issues

* refactor: wrap sqlparser Query inside Query statement variant to reduce complexity

* refactor: replace TryInto to TryFrom

* refactor: use [Rust 2018 mod convention](https://doc.rust-lang.org/edition-guide/rust-2018/path-changes.html#no-more-modrs)

* refactor: remove unnecessary file prefix in statement mod

* feat: implement INSERT parser (currently without INSERT validation)

* feat: wrap DataFusion planner and add simple query planner implementation

* refactor: move planner mod to query crate

* fix: styles and conventions
2022-05-05 16:28:38 +08:00
dennis zhuang
bf331ec4ac feat: impl QueryEngine#execute, adds numbers table and query engine test (#13)
* feat: impl QueryEngine#execute, adds numbers table and query engine test

* fix: clippy warning

* fix: reuse runtime in context in table adapter

* fix: by CR comments
2022-04-27 15:15:26 +08:00
dennis zhuang
3a2f794f6c feat: query engine impl on datafusion (#10)
* feat: query engine impl on datafusion

* feat: adds physical_optimizer, physical_planner and executor

* feat: impl adpaters between datafuion and greptime query engine core APIs.

* feat: impl PhysicalPlanAdapter and ExecutionPlanAdapter

* feat: rename table datafusion mod to adapter

* fix: clippy warning

* fix: conflicts with develop branch

* feat: add database mod

* fix: CR comment

* fix: by CR comments

* fix: conflicts with develop branch

* fix: by CR comments
2022-04-26 15:17:32 +08:00
evenyag
b505c87e8d refactor: Rename base to common-base 2022-04-26 11:48:22 +08:00
evenyag
49081d818c refactor: Divide common crate into small crates 2022-04-25 17:03:05 +08:00
evenyag
d20191572e feat: Implement PrimitiveType and PrimitiveVector 2022-04-25 17:03:05 +08:00
Lei, Huang
cb74f1ac34 feat: Add sql parser definition and trivial SHOW DATABASE implementation (#8)
* feat: Add SQL parser definition and SHOW DATABASE implementation

* chores: Eliminate clippy warnings and errors.

* chores: remove Gt prefix in some structs; rename some mod; remove print(s) in unit tests; refactor crate layout;
feat: wrap sqlparser error;

* chores: reorder cargo dependency

* chores: fix code style

* chores: add #[cfg(test)] to unit tests

* style: fix test mod style
2022-04-25 17:01:55 +08:00
evenyag
1fa0b4e3f9 chore: Setup code skeleton of datanode 2022-04-21 11:40:46 +08:00
evenyag
ced9a7c97f chore: Create datanode, datatypes, log-store, object-store, query, table crates 2022-04-20 16:34:22 +08:00
evenyag
a6c6023737 feat: Init common crate 2022-04-19 18:44:51 +08:00
evenyag
19fa7a6b5b chore: Add Cargo.lock to git 2022-04-19 17:57:50 +08:00