Commit Graph

92 Commits

Author SHA1 Message Date
Jiachun Feng
6cf1da35ee feat: add grpc impl (#50)
* feat: add grpc impl

* feat: add grpc server

* some ut

* verson format: a.b

* code style

* admin request/response

* by cr

* admin api

* by cr

* chore: by cr

* chore: by cr
2022-07-06 20:56:16 +08:00
Lei, Huang
008f62afc1 feat: buffer abstraction (#51)
* feat: add buffer abstraction and rewrite entry encode/decode process

* add some tests

* remove pad.rst

* fix some comments

* fix comments

* remove mmap mod

* feat: Bytes type implementation switch to bytes::Bytes

* fix: use Bytes::from(String) and Bytes::from(Vec<u8>)

* feat: add new method to Entry trait
2022-07-04 14:08:23 +08:00
evenyag
11bf970efd feat: Implement TimestampMillis and RangeMillis (#56) 2022-06-29 20:55:27 +08:00
Lei, Huang
651bdbaa71 fix: log file test fail (#54)
* fix: log file test fail

* remove some log

* wip

* add log

* wip
2022-06-29 17:01:15 +08:00
dennis zhuang
bac6c720f8 feat: impl bytes_allocated for memtable (#55) 2022-06-28 15:11:04 +08:00
dennis zhuang
b567cfb9bc feat: memory size of vector (#53)
* feat: improve try_into_vector function

* feat: impl memory_size function for vectors

* fix: forgot memory_size assertion in null vector test

* feat: use LargeUtf8 instead of utf8 for string, and rename LargeBianryArray to BinaryArray

* feat: memory_size only calculates heap size
2022-06-28 11:06:53 +08:00
dennis zhuang
379d2e2f50 feature: runtime crate and global runtimes (#49)
* feat: init common runtime crate

* feat: tokio Runtime wrapper and global runtime functions

* feat: adds block_on_read, block_on_write, block_on_bg functions to runtime

* refactor: panic when configure global runtimes which are already initialized

* refactor: adds read/write/bg thread pool size

* fix: code style

* fix: clippy warning

* fix: test_metric panic

* fix: address CR problems

* log: adds log when creating runtime
2022-06-21 16:09:15 +08:00
evenyag
6ec870625f refactor: Refactor usage of BoxedError (#48)
* feat: Define a general boxed error

* refactor: common_function use Error in common_query

* feat: Add tests to define_opaque_error macro

* refactor: Refactor table and table engine error

* refactor: recordbatch remove arrow dev-dependency

* refactor: datanode crate use common_error::BoxedError

* chore: Fix clippy

* feat: Returning source status code when using BoxedError

* test: Fix opaque error test

* test: Add tests for table::Error & table_engine::Error

* test: Add test for RecordBatch::new()

* test: Remove generated tests from define_opaque_error

* chore: Address cr comment
2022-06-21 15:24:45 +08:00
dennis zhuang
4071b0cff2 feat: impl scanning data from storage engine for table (#47)
* feat: impl scanning data from storage for MitoTable

* adds test mod to setup table engine test

* fix: comment error

* fix: boyan -> dennis in todo comments

* fix: remove necessary send in BatchIteratorPtr
2022-06-20 15:42:57 +08:00
evenyag
056185eb24 feat(storage): Implement snapshot scan for region (#46)
* feat: Maintain last sequence in VersionControl

* refactor(recordbatch): Replace `Arc<Schema>` by SchemaRef

* feat: Memtable support filter rows with invisible sequence

* feat: snapshot wip

* feat: Implement scan for SnapshotImpl

* test: Add a test that simply puts and scans a region

* chore: Fix clippy

* fix(memtable): Fix memtable returning duplicate keys

* test(memtable): Add sequence visibility test

* test: Add ValueType test

* chore: Address cr comments

* fix: Fix value is not storing but adding to committed sequence
2022-06-20 14:09:31 +08:00
dennis zhuang
e78c015fc0 TableEngine and SqlHandler impl (#45)
* Impl TableEngine, bridge to storage

* Impl sql handler to process insert sql

* fix: minor changes and typo

* test: add datanode test

* test: add table-engine test

* fix: code style

* refactor: split out insert mod from sql and minor changes by CR

* refactor: replace with_context with context
2022-06-17 11:36:49 +08:00
Lei, Huang
e03ac2fc2b Implement log store append and file set management (#43)
* add log store impl

* add some test

* delete failing test

* fix: concurrent close issue

* feat: use arcswap to replace unsafe AtomicPtr

* fix: use lock to protect rolling procedure.
fix: use try_recv to replace poll_recv on appender task.

* chores: 1. use direct tmp dir instead of creating TempDir instance; 2. inline some short function; 3. rename some structs; 4. optimize namespace to arc wrapper inner struct.
2022-06-16 19:09:09 +08:00
fengjiachun
725a261b55 feat(cmd): command refactor (#44)
* feat(cmd): command refactor
2022-06-15 20:08:00 +08:00
fengjiachun
633524709b Merge pull request #42 from GrepTimeTeam/feat/storage/memtable/iter
feat: Add BatchIterator trait and support iterating btree memtable
2022-06-14 17:21:18 +08:00
evenyag
7700a167f2 chores: Address CR comment 2022-06-14 16:53:12 +08:00
evenyag
268598eb57 test: Fix VectorBuilder test and add Value data type test 2022-06-10 17:31:59 +08:00
evenyag
46c5681cb0 chore: Fix clippy 2022-06-10 16:11:07 +08:00
evenyag
9697fbc5e4 test: Add MemtableTester and batch_size test 2022-06-10 15:37:18 +08:00
evenyag
7a55d988fb test: Add simple write/iter test for memtable 2022-06-10 11:53:27 +08:00
evenyag
727bdb8b86 feat: Add sequences and value_types to Batch 2022-06-09 17:30:17 +08:00
evenyag
69b39e7846 feat: Impl BatchIterator for btree memtable
feat: Impl MapIterWrapper

refactor: Rename RowKey to InnerKey
2022-06-09 17:13:02 +08:00
evenyag
4171173b76 feat: Support creating in memory region and writing to memtable (#40)
* chore(store-api): Fix typo in region comments

* feat(storage): Init storage crate

* feat(store-api): Make some method async

* feat(storage): Blank StorageEngine implementation

* feat(storage): StorageEngine returns owned SchemaRef

* feat: pub use arrow in datatypes

* feat(store-api): Implement RegionMetadata

* feat(storage): Impl create region in memory.

* chore(object-store): Format cargo toml

* chore(storage): Log on region created

* feat: Impl CowCell

* feat: Store id to cf meta mapping

* refactor: Refactor version and rename it to VersionControl

* feat: Impl write batch for put, refactor column family

* feat(storage): Skeleton of writing to memtable

* refactor(storage): MemTable returns MemTableSchema

* feat: Add ColumnSchema and conversion between schema and arrow's schema

* feat: Validate put data

* feat: Valid schema of write batch

* feat: insert memtable WIP

* feat: Impl Inserter for memtable

* feat(datatypes): Implement Eq/Ord for Value

feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes

test: Test Value::from()

* feat: Define BTreeMemTable

* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.

* feat: Impl BTreeMemTable::write()

* refactor: Remove useless ColumnFamilyHandle now

* chore: Clean comment

* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value

* test(storage): Add tests for WriteBatch

* chore: Fix clippy

* feat: Add builder for RowKey/ColumnFamilyDescriptor

* test: Add test for metadata

* chore: Fix clippy

* test: Add test for region and engine

* chore: Fix clippy

* chore: Address CR comment
2022-06-09 16:50:02 +08:00
evenyag
8fe577649f feat: Constructing Bytes/StringBytes from Vec<u8>/&[u8]/String/&str (#41) 2022-06-08 14:25:25 +08:00
dennis zhuang
f7136819fc function crate and scalars function (#39)
* feat: adds scalars mod and enhance vectors

* temp commit

* fix compile error

* Impl pow function with new udf framework

* Adds common-function crate and impl scalars function

* fix: remove used code

* test: adds test for function crate and refactor vectors

* fix: fmt style

* feat: impl numpy.clip function

* feat: improve clip function returning int64 type when arguments do not have float type

* feat: adds more test for vectors

* feat: adds replicate method test for primitive vector

* fix: by code review

* feat: clip function returns uint64 when all arguments type are unsigned

* refactor: improve vectors#only_null

* fix: clippy warning

* fix: clippy warning

* fix: clip should return float64 when arguments have both signed and unsigned types
2022-06-08 13:15:22 +08:00
evenyag
23f235524d feat: Implements validity() and null_count() for Vector (#38)
* feat: Add validity() to Vector

* test(datatypes): Add more tests and fix get_data() not returns None for null
2022-06-01 20:55:58 +08:00
Lei, Huang
fb0585229e refactor: Entry should be a trait (#37) 2022-05-26 11:30:50 +08:00
evenyag
383c55d39c ci: Only trigger ci on pull request (#36) 2022-05-25 10:54:43 +08:00
dennis zhuang
a2331366f6 feat: adds adds register_udf api to query engine and refactor datatypes (#34)
* feat: adds ColumnarValue and refactor vectors

* fix: ConcreteDataType compile error

* feat:adds udf/function mods

* feat: adds test for common_query crate

* feat: adds register_udf api to query engine

* feat: adds common_query::error test

* refactor: by CR comments

* refactor: adds impl_new_concret_type_functions! macro to reduce boilerplate codes

* fix: typo
2022-05-24 16:50:56 +08:00
Lei, Huang
06b592f00f feat: add WAL definitions (#35)
* feat: add WAL definitions

* rename and add some tests
2022-05-24 16:12:23 +08:00
evenyag
1594da337f feat(store-api): Prototype of storage engine api (#33) 2022-05-20 18:51:51 +08:00
Lei, Huang
e75a54b766 feat: impl From arrow array for exsisting vectors (#32)
* feat: impl From arrow array for exsisting vectors

* fix: review comments

* feat: clippy forbid prints
2022-05-19 16:10:00 +08:00
dennis zhuang
b0d2e2e91b feat: adds ConcretDataType and more datatypes impl (#31)
* feat: adds ConcretDataType and impl binary/boolean/null types and vectors

* feat: adds String to ConcretDataType

* docs:  ConcretDataType::from_arrow_type may panic
2022-05-19 11:21:11 +08:00
evenyag
5777732fde feat(store-api): Init store-api crate (#30) 2022-05-18 17:19:57 +08:00
Lei, Huang
519cbc832a feat: add StringVector datatype (#28) 2022-05-18 14:49:36 +08:00
Lei, Huang
bd4fe1f5bc feat: RecordBatch serialization (#26) 2022-05-17 17:01:00 +08:00
evenyag
3d374cce68 feat: implement log related macros (#29) 2022-05-17 16:00:17 +08:00
dennis zhuang
a819b6bce5 feat: impl object-store based on opendal (#27)
* feat: impl object-store based on opendal

* feat: adds s3 test

* feat: export Layer

* feat: pass s3 access key/secret by env vars

* fix: logging when s3 test is running
2022-05-17 15:58:27 +08:00
dennis zhuang
93cbdbee9a feat: adds common-telemetry crate and adds logs/metrics to query engine (#25)
* feat: adds common-telemetry crate and logging mod

* refactor: common telemetry

* feat: adds metric mod and exports metrics by http

* feat: adds metrics to query engine and datanodes

* refactor: rename metrics

* refactor: rename ElapsedTimer struct and elapsed_timer macro

* refactor: log error when fail to new data node.

* fix: use backtrace crate instead of unstable feature

* feat: add must_use attr to Timer

* fix: only reserve Debug attribute for Timer
2022-05-12 17:42:48 +08:00
Lei, Huang
641f4b39bd feat: add serialization support to existing data types (#24)
* feat: add serialization support to existing data types

* fix: refactor errors mod to error. fix other review comments
2022-05-11 17:17:27 +08:00
Lei, Huang
841744e4a3 fix: remove direct unwraps (#23) 2022-05-09 17:50:49 +08:00
Lei, Huang
c07784a614 Update codecov.yml
ci: allow coverage drop by 1%
2022-05-09 17:35:44 +08:00
Lei, Huang
5c72c5ae19 tests: add more unit tests for sql mod (#20)
* test: add more unit tests for sql mod
2022-05-09 15:01:19 +08:00
dennis zhuang
df1a99e27f feat: test http server api (#21)
* feat: adds http server test

* feat: limit sql output to 10 rows
2022-05-09 14:54:03 +08:00
dennis zhuang
8cc10b36db Merge pull request #14 from GrepTimeTeam/refactor-error
refactor: Refactor error usage
2022-05-09 14:17:52 +08:00
evenyag
d5de030600 feat: Add as_any() to opaque error 2022-05-09 12:35:55 +08:00
evenyag
5f48b4996b chore: Address CR comments 2022-05-09 11:52:01 +08:00
evenyag
d2d4d88c89 test: Add MockError and add more tests for error 2022-05-07 17:29:56 +08:00
evenyag
e0c1bbfd21 refactor: Rename errors::ParserError to error::Error 2022-05-07 16:15:05 +08:00
evenyag
fae876ec63 test: Add more test for parser error 2022-05-07 16:07:41 +08:00
evenyag
56258d6821 test: Add more test for opaque error 2022-05-07 15:45:46 +08:00