dennis zhuang
c8cb705d9e
ci: pre-commit configuration and hooks ( #261 )
...
* feat: adds pre-commit config and hooks
* refactor: sort all Cargo.toml by cargo-sort
* ci: adds conventional-pre-commit hook to pre-commit
* fix: remove .pre-commit-hooks.yaml
* fix: readme
* Update .pre-commit-config.yaml
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
* ci: move clippy hook to push stage
* docs: install pre-push github hook
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
2022-09-15 11:30:08 +08:00
LFC
ccda17248e
feat: unify servers and mysql server in datanode ( #172 )
...
* address PR comments
address PR comments
use 3306 for mysql server's default port
upgrade metric to version 0.20
move crate "servers" out of "common"
make mysql io threads count configurable in config file
add snafu backtrace for errors with source
use common-server error for mysql server
add test for grpc server
refactor testing codes
fix rustfmt check
start mysql server in datanode
move grpc server codes from datanode to common-servers
feat: unify servers
* rebase develop and resolve conflicts
* remove an unnecessary todo
Co-authored-by: luofucong <luofucong@greptime.com >
2022-08-17 14:29:12 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
dennis zhuang
379d2e2f50
feature: runtime crate and global runtimes ( #49 )
...
* feat: init common runtime crate
* feat: tokio Runtime wrapper and global runtime functions
* feat: adds block_on_read, block_on_write, block_on_bg functions to runtime
* refactor: panic when configure global runtimes which are already initialized
* refactor: adds read/write/bg thread pool size
* fix: code style
* fix: clippy warning
* fix: test_metric panic
* fix: address CR problems
* log: adds log when creating runtime
2022-06-21 16:09:15 +08:00
