Lei, Huang
1dd780d857
feat: implement catalog manager ( #129 )
...
Implement catalog manager that provides a vision of all existing tables while instance start. Current implementation is based on local table engine, all catalog info is stored in an system catalog table.
2022-08-11 15:43:59 +08:00
dennis zhuang
e9d6546c12
feat: impl create_table for MitoEngine, #125 ( #142 )
...
* feat: impl create_table for MitoEngine, #125
* fix: typo
* fix: address CR problems
* fix: address CR problems
* fix: address CR problems
* fix: format
* refactor: minor change
2022-08-08 15:36:00 +08:00
Ning Sun
97be052b33
feat: update tonic/prost and simplify build requirements ( #130 )
...
* feat: update tonic/prost and simplify build requirements
* doc: update readme for protoc installtion
2022-08-04 23:11:39 +08:00
Lei, Huang
7395920bc8
move catalog-related traits and struct to a catalog crate ( #134 )
2022-08-04 11:05:28 +08:00
Lei, Huang
96b4ed01f7
refactor: Make TableEngine object safe ( #119 )
...
* refactor: Make TableEngine object safe
* define TableEngineRef
* fix some comments
* replace table::engine::Error with table::error::Error
2022-08-01 15:37:11 +08:00
fys
3b2716ed70
feat: impl insert via grpc ( #102 )
...
* fix: build protobuf
* feat: impl grpc insert
* Add an example of grpc insert
* fix: cargo clippy
* cr
2022-07-28 10:25:22 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
Jiachun Feng
6cf1da35ee
feat: add grpc impl ( #50 )
...
* feat: add grpc impl
* feat: add grpc server
* some ut
* verson format: a.b
* code style
* admin request/response
* by cr
* admin api
* by cr
* chore: by cr
* chore: by cr
2022-07-06 20:56:16 +08:00
evenyag
6ec870625f
refactor: Refactor usage of BoxedError ( #48 )
...
* feat: Define a general boxed error
* refactor: common_function use Error in common_query
* feat: Add tests to define_opaque_error macro
* refactor: Refactor table and table engine error
* refactor: recordbatch remove arrow dev-dependency
* refactor: datanode crate use common_error::BoxedError
* chore: Fix clippy
* feat: Returning source status code when using BoxedError
* test: Fix opaque error test
* test: Add tests for table::Error & table_engine::Error
* test: Add test for RecordBatch::new()
* test: Remove generated tests from define_opaque_error
* chore: Address cr comment
2022-06-21 15:24:45 +08:00
dennis zhuang
4071b0cff2
feat: impl scanning data from storage engine for table ( #47 )
...
* feat: impl scanning data from storage for MitoTable
* adds test mod to setup table engine test
* fix: comment error
* fix: boyan -> dennis in todo comments
* fix: remove necessary send in BatchIteratorPtr
2022-06-20 15:42:57 +08:00
dennis zhuang
e78c015fc0
TableEngine and SqlHandler impl ( #45 )
...
* Impl TableEngine, bridge to storage
* Impl sql handler to process insert sql
* fix: minor changes and typo
* test: add datanode test
* test: add table-engine test
* fix: code style
* refactor: split out insert mod from sql and minor changes by CR
* refactor: replace with_context with context
2022-06-17 11:36:49 +08:00
fengjiachun
725a261b55
feat(cmd): command refactor ( #44 )
...
* feat(cmd): command refactor
2022-06-15 20:08:00 +08:00
Lei, Huang
bd4fe1f5bc
feat: RecordBatch serialization ( #26 )
2022-05-17 17:01:00 +08:00
dennis zhuang
93cbdbee9a
feat: adds common-telemetry crate and adds logs/metrics to query engine ( #25 )
...
* feat: adds common-telemetry crate and logging mod
* refactor: common telemetry
* feat: adds metric mod and exports metrics by http
* feat: adds metrics to query engine and datanodes
* refactor: rename metrics
* refactor: rename ElapsedTimer struct and elapsed_timer macro
* refactor: log error when fail to new data node.
* fix: use backtrace crate instead of unstable feature
* feat: add must_use attr to Timer
* fix: only reserve Debug attribute for Timer
2022-05-12 17:42:48 +08:00
dennis zhuang
df1a99e27f
feat: test http server api ( #21 )
...
* feat: adds http server test
* feat: limit sql output to 10 rows
2022-05-09 14:54:03 +08:00
evenyag
d5de030600
feat: Add as_any() to opaque error
2022-05-09 12:35:55 +08:00
evenyag
5f48b4996b
chore: Address CR comments
2022-05-09 11:52:01 +08:00
evenyag
d2d4d88c89
test: Add MockError and add more tests for error
2022-05-07 17:29:56 +08:00
evenyag
99c7ffb456
refactor: Define error for memory catalog
2022-05-07 14:13:31 +08:00
evenyag
63d9aa1bff
refactor: Refactor datanode error and impl ErrorExt for it
2022-05-07 11:34:23 +08:00
evenyag
aa1a6b2205
feat: Add common-error crate and implement opaque error type.
2022-05-06 20:04:14 +08:00
dennis zhuang
f0db878bf5
feat: http server and cmd crate etc. ( #15 )
...
* feat: adds cmd crate and http server
* feat: impl sql http handler
* feat: convert all arrow array types
* feat: adds query test
* feat: adds test for datanode
* fix: format
* feat: refactor state.rs
* feat: adds collect test
* fix: by code review
* fix: style
2022-05-06 17:34:29 +08:00
evenyag
1fa0b4e3f9
chore: Setup code skeleton of datanode
2022-04-21 11:40:46 +08:00
evenyag
ced9a7c97f
chore: Create datanode, datatypes, log-store, object-store, query, table crates
2022-04-20 16:34:22 +08:00
