* refactor: return PhysicalPlan in Table trait's scan method, to support partitioned execution in Frontend's distribute read
* refactor: pub use necessary DataFusion types
* refactor: replace old "PhysicalPlan" and its adapters
Co-authored-by: luofucong <luofucong@greptime.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
* feat: add type conversion optimizer
* feat: add expr rewrite logical plan optimizer
* chore: add some doc
* fix: unit test
* fix: time zone issue in unit tests
* chore: add more tests
* fix: some CR comments
* chore: rebase develop
* chore: fix unit tests
* fix: unit test use timestamp with time zone
* chore: add more tests
* feat: ensure time index column can't be included in primary key
* feat: sql parser supports show tables statement
* feat: impl show databases and show tables, #183
* feat: impl like expression for show databases/tables and add tests
* fix: typo
* fix: address CR problems
* wip add predicate definition
* fix value move
* implement predicate and prune
* impl filter push down in chunk reader
* add more expr tests
* chore: rebase develop
* fix: unit test
* fix: field name/index lookup when building pruning stats
* chore: add some meaningless test
* fix: remove unnecessary extern crate
* fix: use datatypes::schema::SchemaRef
* feat: frontend instance
* no need to carry column length in `Column` proto
* add more tests
* rebase develop
* create a new variant with already provisioned RecordBatches in Output
* resolve code review comments
* new frontend instance does not connect datanode grpc
* add more tests
* add more tests
* rebase develop
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: Impl cmp_element() for Vector
* chore: Add doc comments to MutableVector
* feat: Add create_mutable() to DataType
Add `create_mutable()` to create a MutableVector for each DataType.
Implement ListVectorBuilder and NullVectorBuilder for ListType and
NullType.
* feat: Add ValueRef
ValueRef is a reference to value, could be used to avoid some allocation
when getting data from Vector. To support ValueRef, also implement a
ListValueRef for ListValue, but comparision of ListValueRef still
requires some allocation, due to the complexity of ListValue and
ListVector.
Impl some From trait for ValueRef
* feat: Implement get_ref for Vector
* feat: Remove cmp_element from Vector
`cmp_element` could be replaced by `get_ref` and then compare
* feat: Implement push/extend for PrimitiveVectorBuilder
Implement push_value_ref() and extend_slice_of() for
PrimitiveVectorBuilder.
Also refactor the DataTypeBuilder trait for
primitive types to PrimitiveElement trait, adds necessary cast helper
methods to it.
- Cast a reference to Vector to reference arrow's primitive array
- Cast a ValueRef to primitive type
- Also make PrimitiveElement super trait of Primitive
* feat: Implement push/extend for all vector builders
Implement push_value_ref() and extend_slice_of() for remaining vector
builders. Add some helpful cast method to ValueRef and a method to
cast Value to ValueRef.
Change the behavior of PrimitiveElement::cast_xxx to panic when unable
to cast, since push_value_ref() and extend_slice_of() always panic
when given invalid input data type.
* feat: MutableVector returns error if data type unmatch
* test: Add tests for ValueRef
* feat: Add tests for Vector::get_ref
* feat: NullVector returns error if data type unmatch
* test: Add tests for vector builders
* fix: Fix compile error in python coprocessor
* refactor: Add lifetime param to IntoValueRef
The Primitive trait just use the `IntoValueRef<'static>` bound. Also
rename create_mutable to create_mutable_vector.
* chore: Address CR comments
* feat: Customize PartialOrd/Ord for Value/ValueRef
Panics if values/refs have different data type
* style: Fix clippy
* refactor: Use macro to generate body of ValueRef::as_xxx
* feat: implement alter table
* Currently we have no plans to support altering the primary keys (maybe never), so removed the related codes.
* make `alter` a trait function in table
* address other CR comments
* cleanup
* rebase develop
* resolve code review comments
Co-authored-by: luofucong <luofucong@greptime.com>
* address PR comments
address PR comments
use 3306 for mysql server's default port
upgrade metric to version 0.20
move crate "servers" out of "common"
make mysql io threads count configurable in config file
add snafu backtrace for errors with source
use common-server error for mysql server
add test for grpc server
refactor testing codes
fix rustfmt check
start mysql server in datanode
move grpc server codes from datanode to common-servers
feat: unify servers
* rebase develop and resolve conflicts
* remove an unnecessary todo
Co-authored-by: luofucong <luofucong@greptime.com>
Implement catalog manager that provides a vision of all existing tables while instance start. Current implementation is based on local table engine, all catalog info is stored in an system catalog table.
* feat: memtable backed by DataFusion to ease testing
* move test utility codes out of src folder
* Implement our own MemTable because DataFusion's MemTable does not support limit; and replace the original testing numbers table.
* fix: address PR comments
* fix: "testutil" -> "test-util"
* roll back "NumbersTable"
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: UDAF implementation backed by DataFusion.
Directly Transplant DataFusion's UDAF related structs, traits and functions, like `AggregateUDF`, `Accumulator` or `create_udaf` etc.
Implement median UDAF on top of it and used in unit testing.
Refs: #61
* feat: UDAF made generically
Refs: #61
* fix: cargo fmt
* fix: use prelude
* fix: uniform the name
* fix: move maybe commonly used functions together
* fix: make comments more clear
* fix: resolve conversations in CR
* fix: store input types in AccumulatorCreator, and use ScalarVector's iterator
* feat: introducing List value and List datatype
* refactor: use ArcSwap instead of Mutext
* refactor: shorten some namings
* refactor: move median UDAF out of tests
* refactor: rename
* feat: aggregate function registry
* fix: make `Value` satisfy ordering again
* fix: clippy warnings
* doc: add "how to write aggregate function"
* fix: address PR comments
* fix: trying to get rid of unwraps
Co-authored-by: luofucong <luofucong@greptime.com>
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* Impl TableEngine, bridge to storage
* Impl sql handler to process insert sql
* fix: minor changes and typo
* test: add datanode test
* test: add table-engine test
* fix: code style
* refactor: split out insert mod from sql and minor changes by CR
* refactor: replace with_context with context
* chore(store-api): Fix typo in region comments
* feat(storage): Init storage crate
* feat(store-api): Make some method async
* feat(storage): Blank StorageEngine implementation
* feat(storage): StorageEngine returns owned SchemaRef
* feat: pub use arrow in datatypes
* feat(store-api): Implement RegionMetadata
* feat(storage): Impl create region in memory.
* chore(object-store): Format cargo toml
* chore(storage): Log on region created
* feat: Impl CowCell
* feat: Store id to cf meta mapping
* refactor: Refactor version and rename it to VersionControl
* feat: Impl write batch for put, refactor column family
* feat(storage): Skeleton of writing to memtable
* refactor(storage): MemTable returns MemTableSchema
* feat: Add ColumnSchema and conversion between schema and arrow's schema
* feat: Validate put data
* feat: Valid schema of write batch
* feat: insert memtable WIP
* feat: Impl Inserter for memtable
* feat(datatypes): Implement Eq/Ord for Value
feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes
test: Test Value::from()
* feat: Define BTreeMemTable
* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.
* feat: Impl BTreeMemTable::write()
* refactor: Remove useless ColumnFamilyHandle now
* chore: Clean comment
* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value
* test(storage): Add tests for WriteBatch
* chore: Fix clippy
* feat: Add builder for RowKey/ColumnFamilyDescriptor
* test: Add test for metadata
* chore: Fix clippy
* test: Add test for region and engine
* chore: Fix clippy
* chore: Address CR comment
* feat: adds scalars mod and enhance vectors
* temp commit
* fix compile error
* Impl pow function with new udf framework
* Adds common-function crate and impl scalars function
* fix: remove used code
* test: adds test for function crate and refactor vectors
* fix: fmt style
* feat: impl numpy.clip function
* feat: improve clip function returning int64 type when arguments do not have float type
* feat: adds more test for vectors
* feat: adds replicate method test for primitive vector
* fix: by code review
* feat: clip function returns uint64 when all arguments type are unsigned
* refactor: improve vectors#only_null
* fix: clippy warning
* fix: clippy warning
* fix: clip should return float64 when arguments have both signed and unsigned types
* feat: adds common-telemetry crate and logging mod
* refactor: common telemetry
* feat: adds metric mod and exports metrics by http
* feat: adds metrics to query engine and datanodes
* refactor: rename metrics
* refactor: rename ElapsedTimer struct and elapsed_timer macro
* refactor: log error when fail to new data node.
* fix: use backtrace crate instead of unstable feature
* feat: add must_use attr to Timer
* fix: only reserve Debug attribute for Timer
