* address PR comments
address PR comments
use 3306 for mysql server's default port
upgrade metric to version 0.20
move crate "servers" out of "common"
make mysql io threads count configurable in config file
add snafu backtrace for errors with source
use common-server error for mysql server
add test for grpc server
refactor testing codes
fix rustfmt check
start mysql server in datanode
move grpc server codes from datanode to common-servers
feat: unify servers
* rebase develop and resolve conflicts
* remove an unnecessary todo
Co-authored-by: luofucong <luofucong@greptime.com>
* fix: Rename current_timestamp to current_time_millis, fix resolution
Fix current_timestamp returns seconds resolution, also add a test for
this method
* chore: Use slice of array instead of Vec
Save some heap allocations
* test: Compare std and chrono timestamp
The original test always success even the current_time_millis returns in
seconds resolution
* chore: Store current time in gmt_created/gmt_modified
* catalog manager allocates table id
* rebase develop
* add some tests
* add some more test
* fix some cr comments
* insert into system catalog
* use slice pattern to simplify code
* add optional dependencies
* add sql-to-request test
* successfully recover
* fix unit tests
* rebase develop
* add some tests
* fix some cr comments
* fix some cr comments
* add a lock to CatalogManager
* feat: add gmt_created and gmt_modified columns to system catalog table
* SelectExpr: change to oneof expr
* Convert between Vec<u8> and SelectResult
* Chore: use encode_to_vec and decode, instead of encode_length_delimited_to_vec and decode_length_delimited
* Chore: move bitset into separate file
* Grpc select impl
* feat: impl TableManifest and refactor table engine, object store etc.
* feat: persist table metadata when creating it
* fix: remove unused file src/storage/src/manifest/impl.rs
* feat: impl recover table info from manifest
* test: add open table test and table manifest test
* fix: resolve CR problems
* fix: compile error and remove region id
* doc: describe parent_dir
* fix: address CR problems
* fix: typo
* Revert "fix: compile error and remove region id"
This reverts commit c14c250f8a.
* fix: compile error and generate region id by table_id and region number
Implement catalog manager that provides a vision of all existing tables while instance start. Current implementation is based on local table engine, all catalog info is stored in an system catalog table.
* add pwrite
* write
* fix write
* error handling in write thread
* wrap some LogFile field to state field
* remove some unwraps
* reStructure some code
* implement file chunk
* composite chunk decode
* add test for chunk stream
* fix buffer test
* remove some useless code
* add test for read_at and file_chunk_stream
* use bounded channel to implement back pressure
* reimplement entry read and decoding
* add some doc
* clean some code
* use Sender::blocking_send to replace manually spawn
* support synchronous file chunk stream
* remove useless clone
* remove set_offset from Entry trait
* cr: fix some comments
* fix: add peek methods for Buffer
* add test for read at the middle of file
* fix some minor issues on comments
* rebase on to develop
* add peek_to_slice and read_to_slice
* initialize file chunk on heap
* fix some comments in CR
* respect entry id set outside LogStore
* fix unit test
* Update src/log-store/src/fs/file.rs
Co-authored-by: evenyag <realevenyag@gmail.com>
* fix some cr comments
Co-authored-by: evenyag <realevenyag@gmail.com>
* feat: protobuf codec
* chore: minor fix
* chore: beatify the macro code
* chore: minor fix
* chore: by cr
* chore: by cr and impl wal with proto
* bugfix: invalid num_rows for multi put_data in mutations
Co-authored-by: jiachun <jiachun_fjc@163.com>
* refactor(storage): Add region id and name to metadata
Add region id and name to `RegionMetadata`, simplify input arguments of
`RegionImpl::create()` and `RegionImpl::new()` method, since id and name
are already in metadata/version.
To avoid an atomic load of `Version` each time we access the region
id/name, we still store a copy of id/name in `SharedData`.
* chore: Remove todo in OpenOptions
Create region if missing when opening the region would be hard to
implement, since sometimes we may don't known the exact region schema user
would like to have.
* refactor: Make id and name of region readonly
By making `id` and `name` fields of `SharedData` and `RegionMetadata`
private and only exposing a pub getter.
* feat: memtable backed by DataFusion to ease testing
* move test utility codes out of src folder
* Implement our own MemTable because DataFusion's MemTable does not support limit; and replace the original testing numbers table.
* fix: address PR comments
* fix: "testutil" -> "test-util"
* roll back "NumbersTable"
Co-authored-by: luofucong <luofucong@greptime.com>
* feat: Add `open_table()` method to `TableEngine`
* feat: Implements MitoEngine::open_table()
For simplicity, this implementation just use the table name as region
name, and using that name to open a region for that table. It also
introduce a mutex to avoid opening the same table simultaneously.
* refactor: Shorten generic param name
Use `S` instead of `Store` for `MitoEngine`.
* test: Mock storage engine for table engine test
Add a `MockEngine` to mock the storage engine, so that testing the mito
table engine can sometimes use the mocked storage.
* test: Add open table test
Also remove `storage::gen_region_name` method, and always use table name
as default region name, so the table engine can open the table created
by `create_table()`.
* chore: Add open table log
* feat: Implements RegionWriter::replay()
Refactors `preprocess_write()`, wraps time ranges calculation and
memtable creation to `prepare_memtables()` so these logic can be reused
by `WriterInner::replay()`. Then implements `WriterInner::replay()`
which reads write batch from wal and inserts it into memtables.
* feat: Use sequence in request as committed sequence
Also checks that sequence should increase monotonically and returns
error if found sequence decreases
* chore: Remove OpenOptions param from RegionWriter::replay
* test: Add region reopen tests
refactor(storage): Rename read_write test mod to basic
refactor(storage): Move common region test logic to TesterBase
Let read/write Tester and flush Tester share the same TesterBase struct,
which implements common operations like put/full_scan.
* feat: Constructs RegionImpl in open()
Constructs RegionImpl after replay in `RegionImpl::open()`
* feat: Adds RegionImpl::create()
Adds `RegionImpl::create()` method to persist region metadata to
manifest, then create the RegionImpl instance, so the storage engine
just invoke `RegionImpl::create()` instead of `RegionImpl::new()` to
create the region instance, and don't need to update manifest after
creating region instance anymore. Now `RegionImpl::new()` need to takes
version instead of metadata as input.
This change is also a necessary part to pass the region open test, since
to open a region, need to persist something to manifest first.
* feat: Pass region open test
Use LocalFileLogStore for region test since NoopLogStore won't persist
data to the file system.
Create dir in `LocalFileLogStore::open` if it is not exist, so we don't
need to create the dir before using the logstore.
To pass the test, we always recover from flushed_sequence and use
`req_sequence + 1` as last sequence.
* test: Test reopen region multiple times
* chore: Address CR comments
Add more info to replay log and add an assert to check committed
sequence after reopen.
* refactor: Add cfg(test) to Version::new()
Remove `VersionControl::new()`, and add `#[cfg(test)]` to
`Version::new()` as it is only used by tests.
* feat: impl recovering version from manifest for region
* refactor: rename try_apply_edit to replay_edit
* fix: remove println
* fix: address CR problems
* feat: remove Metadata in manifest trait and update region manifest state after recovering
* refactor: Move fields in SharedData to EngineInner
Since `SharedData` isn't shared now, we move all its fields to
EngineInner, and remove the `SharedData` struct, also remove the
unused config field.
* feat: Store RegionSlot in engine's region map
A `RegionSlot` has three possible state:
- Opening
- Creating
- Ready (Holds the `RegionImpl`)
Also use the `RegionSlot` as a placeholder in the region map to indicate
the region is opening/creating, so another open/create request will
fail immediately. The `SlotGuard` is used to clean the slot if we failed
to create/open the region.
* feat: Add a blank method `RegionImpl::open`
* feat: Remove MetadataId from Manifest
Now metadata id of manifest is unused, also unnecessary as we have
manifest dir to build the manifest, but constructing the manifest
still needs a passing region id as argument, which is unavailable
during opening region. So we remove the metadata id from manifest so
`region_store_config()` don't need region id as input anymore
* feat: Remove region id from logstore::Namespace and Wal
This is necessary for implementing open, since we don't have region
id this time, but we need to build Wal and its logstore namespace. Now
this is ok as id is not actually used by logstore.
* feat: Setup `open_region` code skeleton
* fix: Fix write stall blocks flush applying version
refactor: Use store config to help constructing Region
chore: Address CR comments
* feat: adds manifest protocol supporting and refactor region metadata protocol
feat: ignore sqlparser log
refactor: PREV_VERSION_KEY constant
refactor: minor change for checking readable/writable
fix: address CR problems
refactor: use binary literal
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
* feat: UDAF implementation backed by DataFusion.
Directly Transplant DataFusion's UDAF related structs, traits and functions, like `AggregateUDF`, `Accumulator` or `create_udaf` etc.
Implement median UDAF on top of it and used in unit testing.
Refs: #61
* feat: UDAF made generically
Refs: #61
* fix: cargo fmt
* fix: use prelude
* fix: uniform the name
* fix: move maybe commonly used functions together
* fix: make comments more clear
* fix: resolve conversations in CR
* fix: store input types in AccumulatorCreator, and use ScalarVector's iterator
* feat: introducing List value and List datatype
* refactor: use ArcSwap instead of Mutext
* refactor: shorten some namings
* refactor: move median UDAF out of tests
* refactor: rename
* feat: aggregate function registry
* fix: make `Value` satisfy ordering again
* fix: clippy warnings
* doc: add "how to write aggregate function"
* fix: address PR comments
* fix: trying to get rid of unwraps
Co-authored-by: luofucong <luofucong@greptime.com>
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* move interp from test to numpy
* feat: implement InsertExpr
* 1.InsertExpr reverted to previous version 2.add InsertBatch message
* add two SemanticTypes: TAG, TIMESTAMP
* chore: format proto files
* chore: add some comments about "Column"
* fix: rename "semanticType" -> "semantic_type"
* fix: unique number in InsertBatch
* fix: type of f64_values
* chore: move insertbatch and column to insert.proto
* chore: rename "ExprHeader" to "Header"
* fix: ExprHeader not found in this scope
* feat: add grpc impl
* feat: add grpc server
* some ut
* verson format: a.b
* code style
* admin request/response
* by cr
* admin api
* by cr
* chore: by cr
* chore: by cr
