Commit Graph

22 Commits

Author SHA1 Message Date
LFC
ccda17248e feat: unify servers and mysql server in datanode (#172)
* address PR comments

address PR comments

use 3306 for mysql server's default port

upgrade metric to version 0.20

move crate "servers" out of "common"

make mysql io threads count configurable in config file

add snafu backtrace for errors with source

use common-server error for mysql server

add test for grpc server

refactor testing codes

fix rustfmt check

start mysql server in datanode

move grpc server codes from datanode to common-servers

feat: unify servers

* rebase develop and resolve conflicts

* remove an unnecessary todo

Co-authored-by: luofucong <luofucong@greptime.com>
2022-08-17 14:29:12 +08:00
LFC
4098c57446 feat: MySQL protocol server (#158)
* MySQL protocol server

* fix: Rustfmt check

* fix: resolve PR comments

Co-authored-by: luofucong <luofucong@greptime.com>
2022-08-12 11:41:45 +08:00
LFC
e833167ad6 feat: extract MemTable to ease testing (#133)
* feat: memtable backed by DataFusion to ease testing

* move test utility codes out of src folder

* Implement our own MemTable because DataFusion's MemTable does not support limit; and replace the original testing numbers table.

* fix: address PR comments

* fix: "testutil" -> "test-util"

* roll back "NumbersTable"

Co-authored-by: luofucong <luofucong@greptime.com>
2022-08-05 13:58:05 +08:00
Lei, Huang
7395920bc8 move catalog-related traits and struct to a catalog crate (#134) 2022-08-04 11:05:28 +08:00
evenyag
bf5975ca3e feat: Prototype of the storage engine (#107)
* feat: memtable flush (#63)

* wip: memtable flush

* optimize schema conversion

* remove unnecessary import

* add parquet file verfication

* add backtrace to error

* chore: upgrade opendal to 0.9 and fixed some problems

* rename error

* fix: error description

Co-authored-by: Dennis Zhuang <killme2008@gmail.com>

* feat: region manifest service (#57)

* feat: adds Manifest API

* feat: impl region manifest service

* refactor: by CR comments

* fix: storage error mod test

* fix: tweak storage cargo

* fix: tweak storage cargo

* refactor: by CR comments

* refactor: rename current_version

* feat: add wal writer (#60)

* feat: add Wal

* upgrade engine for wal

* fix: unit test for wal

* feat: wal into region

* fix: unix test

* fix clippy

* chore: by cr

* chore: by cr

* chore: prevent test data polution

* chore: by cr

* minor fix

* chore: by cr

* feat: Implement flush (#65)

* feat: Flush framework

- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports

refactor: Use ParquetWriter to replace FlushTask

refactor: FsAccessLayer takes object store as param

chore: Remove todo from doc comments

feat: Move wal to WriterContext

chore: Fix clippy

chore: Add backtrace to WriteWal error

* feat: adds manifest to region and refactor sst/manifest dir config (#72)

* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig

* refactor: ensure path ends with '/' in ManifestLogStorage

* fix: style

* refactor: normalize storage directory path and minor changes by CR

* refactor: doesn't need slash any more

* feat: Implement apply_edit() and add timestamp index to schema (#73)

* feat: Implement VersionControl::apply_edit()

* feat: Add timestamp index to schema

* feat: Implement Schema::timestamp_column()

* feat: persist region metadata to manifest (#74)

* feat: persist metadata when creating region or sst files

* fix: revert FileMeta comment

* feat: resolve todo

* fix: clippy warning

* fix: revert files_to_remove type in RegionEdit

* feat: impl SizeBasedStrategy for flush (#76)

* feat: impl SizeBasedStrategy for flush

* doc: get_mutable_limitation

* fix: code style and comment

* feat: align timestamp (#75)

* feat: align timestamps in write batch

* fix cr comments

* fix timestamp overflow

* simplify overflow check

* fix cr comments

* fix clippy issues

* test: Fix region tests (comment out some unsupported tests) (#82)

* feat: flush job (#80)

* feat: flush job

* fix cr comments

* move file name instead of clone

* comment log file test (#84)

* feat: improve MemtableVersion (#78)

* feat: improve MemtableVersion

* feat: remove flushed immutable memtables and test MemtableVersion

* refactor: by CR comments

* refactor: clone kv in iterator

* fix: clippy warning

* refactor: Make BatchIterator supertrait of Iterator (#85)

* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83)

* feat: Insert multiple memtables by time range (#77)

* feat: memtable::Inserter supports insert multiple memtables by time range

* chore: Update timestamp comment

* test: Add tests for Inserter

* test: Fix region tests (comment out some unsupported tests)

* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()

* chore: rename aligned_by_bucket to align_by_bucket

* fix: Fix compile errors

* fix: sst and manifest dir (#86)

* Set RowKeyDescriptor::enable_version_column to false by default

* feat: Implement write stall (#90)

* feat: Implement write stall

* chore: Update comments

* feat: Support reading multiple memtables (#93)

* feat: Support reading multiple memtables

* test: uncomment tests rely on snapshot read

* feat: wal format (#70)

* feat: wal codec

* chore: minor fix

* chore: comment

* chore: by cr

* chore: write_batch_codec mod

* chore: by cr

* chore: upgrade proto

* chore: by cr

* fix failing test

* fix failing test

* feat: manifest to wal (#100)

* feat: write manifest to wal

* chore: sequence into wal

* chore: by cr

* chore: by cr

* refactor: create log store (#104)

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: fariygirl <clickmetoday@163.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: Fix clippy

Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
Co-authored-by: fariygirl <clickmetoday@163.com>
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>
2022-07-25 15:26:00 +08:00
Jiachun Feng
6cf1da35ee feat: add grpc impl (#50)
* feat: add grpc impl

* feat: add grpc server

* some ut

* verson format: a.b

* code style

* admin request/response

* by cr

* admin api

* by cr

* chore: by cr

* chore: by cr
2022-07-06 20:56:16 +08:00
evenyag
11bf970efd feat: Implement TimestampMillis and RangeMillis (#56) 2022-06-29 20:55:27 +08:00
dennis zhuang
379d2e2f50 feature: runtime crate and global runtimes (#49)
* feat: init common runtime crate

* feat: tokio Runtime wrapper and global runtime functions

* feat: adds block_on_read, block_on_write, block_on_bg functions to runtime

* refactor: panic when configure global runtimes which are already initialized

* refactor: adds read/write/bg thread pool size

* fix: code style

* fix: clippy warning

* fix: test_metric panic

* fix: address CR problems

* log: adds log when creating runtime
2022-06-21 16:09:15 +08:00
dennis zhuang
e78c015fc0 TableEngine and SqlHandler impl (#45)
* Impl TableEngine, bridge to storage

* Impl sql handler to process insert sql

* fix: minor changes and typo

* test: add datanode test

* test: add table-engine test

* fix: code style

* refactor: split out insert mod from sql and minor changes by CR

* refactor: replace with_context with context
2022-06-17 11:36:49 +08:00
evenyag
4171173b76 feat: Support creating in memory region and writing to memtable (#40)
* chore(store-api): Fix typo in region comments

* feat(storage): Init storage crate

* feat(store-api): Make some method async

* feat(storage): Blank StorageEngine implementation

* feat(storage): StorageEngine returns owned SchemaRef

* feat: pub use arrow in datatypes

* feat(store-api): Implement RegionMetadata

* feat(storage): Impl create region in memory.

* chore(object-store): Format cargo toml

* chore(storage): Log on region created

* feat: Impl CowCell

* feat: Store id to cf meta mapping

* refactor: Refactor version and rename it to VersionControl

* feat: Impl write batch for put, refactor column family

* feat(storage): Skeleton of writing to memtable

* refactor(storage): MemTable returns MemTableSchema

* feat: Add ColumnSchema and conversion between schema and arrow's schema

* feat: Validate put data

* feat: Valid schema of write batch

* feat: insert memtable WIP

* feat: Impl Inserter for memtable

* feat(datatypes): Implement Eq/Ord for Value

feat: Implement Ord/Eq for Bytes/StringBytes and Deref for Bytes

test: Test Value::from()

* feat: Define BTreeMemTable

* Fix: Rename get/get_unchecked to try_get/get and fix get not consider null.

* feat: Impl BTreeMemTable::write()

* refactor: Remove useless ColumnFamilyHandle now

* chore: Clean comment

* feat(common): Add from `String/&str/Vec<u8>/&[u8]` for Value

* test(storage): Add tests for WriteBatch

* chore: Fix clippy

* feat: Add builder for RowKey/ColumnFamilyDescriptor

* test: Add test for metadata

* chore: Fix clippy

* test: Add test for region and engine

* chore: Fix clippy

* chore: Address CR comment
2022-06-09 16:50:02 +08:00
dennis zhuang
f7136819fc function crate and scalars function (#39)
* feat: adds scalars mod and enhance vectors

* temp commit

* fix compile error

* Impl pow function with new udf framework

* Adds common-function crate and impl scalars function

* fix: remove used code

* test: adds test for function crate and refactor vectors

* fix: fmt style

* feat: impl numpy.clip function

* feat: improve clip function returning int64 type when arguments do not have float type

* feat: adds more test for vectors

* feat: adds replicate method test for primitive vector

* fix: by code review

* feat: clip function returns uint64 when all arguments type are unsigned

* refactor: improve vectors#only_null

* fix: clippy warning

* fix: clippy warning

* fix: clip should return float64 when arguments have both signed and unsigned types
2022-06-08 13:15:22 +08:00
evenyag
5777732fde feat(store-api): Init store-api crate (#30) 2022-05-18 17:19:57 +08:00
dennis zhuang
93cbdbee9a feat: adds common-telemetry crate and adds logs/metrics to query engine (#25)
* feat: adds common-telemetry crate and logging mod

* refactor: common telemetry

* feat: adds metric mod and exports metrics by http

* feat: adds metrics to query engine and datanodes

* refactor: rename metrics

* refactor: rename ElapsedTimer struct and elapsed_timer macro

* refactor: log error when fail to new data node.

* fix: use backtrace crate instead of unstable feature

* feat: add must_use attr to Timer

* fix: only reserve Debug attribute for Timer
2022-05-12 17:42:48 +08:00
evenyag
aa1a6b2205 feat: Add common-error crate and implement opaque error type. 2022-05-06 20:04:14 +08:00
dennis zhuang
f0db878bf5 feat: http server and cmd crate etc. (#15)
* feat: adds cmd crate and http server

* feat: impl sql http handler

* feat: convert all arrow array types

* feat: adds query test

* feat: adds test for datanode

* fix: format

* feat: refactor state.rs

* feat: adds collect test

* fix: by code review

* fix: style
2022-05-06 17:34:29 +08:00
dennis zhuang
3a2f794f6c feat: query engine impl on datafusion (#10)
* feat: query engine impl on datafusion

* feat: adds physical_optimizer, physical_planner and executor

* feat: impl adpaters between datafuion and greptime query engine core APIs.

* feat: impl PhysicalPlanAdapter and ExecutionPlanAdapter

* feat: rename table datafusion mod to adapter

* fix: clippy warning

* fix: conflicts with develop branch

* feat: add database mod

* fix: CR comment

* fix: by CR comments

* fix: conflicts with develop branch

* fix: by CR comments
2022-04-26 15:17:32 +08:00
evenyag
49081d818c refactor: Divide common crate into small crates 2022-04-25 17:03:05 +08:00
evenyag
1fa0b4e3f9 chore: Setup code skeleton of datanode 2022-04-21 11:40:46 +08:00
evenyag
ced9a7c97f chore: Create datanode, datatypes, log-store, object-store, query, table crates 2022-04-20 16:34:22 +08:00
evenyag
a6c6023737 feat: Init common crate 2022-04-19 18:44:51 +08:00
evenyag
14b0f543be chore: Rename package name to greptime 2022-04-19 17:57:50 +08:00
dennis zhuang
4ca5387bb8 chore: Initial commit 2022-04-19 17:56:27 +08:00