LFC
8fef32f8ef
feat: enable tokio console in cluster mode ( #1512 )
...
* feat: enable tokio console subscriber
* fix: resolve PR comments
* fix: resolve PR comments
* fix: resolve PR comments
2023-05-11 14:35:06 +08:00
Yingwen
1833e487a4
refactor: remove unnecessary async from RepeatedTask::start ( #1545 )
...
* refactor: relax RepeatedTask requirements
Some refactor:
- Remove async from start()
- Cancel task in drop
- Allow TaskFunction::call taking &mut self
- Make start/stop concurrent safe
* test(log-store): Fix log store tests (start multiple times)
2023-05-09 21:03:15 +08:00
localhost
0c88bb09e3
chore: add some metrics for grpc client ( #1398 )
...
* chore: add some metrics for grpc client
* chore: add grpc preix and change metrics-exporter-ptometheus to add global prefix
---------
Co-authored-by: paomian <qtang@greptime.com >
2023-04-18 13:55:01 +08:00
Ruihang Xia
da66138e80
refactor(error): remove backtrace, and introduce call-site location for debugging ( #1329 )
...
* wip: global replace
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix compile
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove unneeded tests of errors
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ErrorExt trait implementator
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix warnings
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix pyo3 tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-04-06 04:06:00 +00:00
localhost
a2b262ebc0
chore: add http metrics server in datanode node when greptime start in distributed mode ( #1256 )
...
* chore: add http metrics server in datanode node when greptime start in distributed mode
* chore: add some docs and license
* chore: change metrics_addr to resolve address already in use error
* chore add metrics for meta service
* chore: replace metrics exporter http server from hyper to axum
* chore: format
* fix: datanode mode branching error
* fix: sqlness test address already in use and start metrics in defualt config
* chore: change metrics location
* chore: use builder pattern to builder httpserver
* chore: remove useless debug_assert macro in httpserver builder
* chore: resolve conflicting build error
* chore: format code
2023-03-31 18:37:52 +08:00
dennis zhuang
972f64c3d7
chore: improve opendal layers ( #1295 )
...
* chore: improve opendal layers
* chore: log level
2023-03-31 09:48:11 +00:00
dennis zhuang
563adbabe9
feat!: improve region manifest service ( #1268 )
...
* feat: try to use batch delete in ManifestLogStorage
* feat: clean temp dir when startup with file backend
* refactor: export region manifest checkpoint actions magin and refactor storage options
* feat: purge unused manifest and checkpoint files by repeat gc task
* chore: debug deleted logs
* feat: adds RepeatedTask and refactor all gc tasks
* chore: clean code
* feat: export gc_duration to manifest config
* test: assert gc works
* fix: typo
* Update src/common/runtime/src/error.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: LFC <bayinamine@gmail.com >
* fix: format
* Update src/common/runtime/src/repeated_task.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: by CR comments
* chore: by CR comments
* fix: serde default for StorageConfig
* chore: remove compaction config in StandaloneOptions
---------
Co-authored-by: LFC <bayinamine@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-03-31 10:42:00 +08:00
Lei, HUANG
8ffc078f88
fix: license header ( #815 )
2023-01-03 15:09:49 +08:00
Ruihang Xia
26a3e93ca7
chore: util workspace deps in more places ( #792 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-12-27 16:26:59 +08:00
LFC
dc52a51576
chore: upgrade to Arrow 29.0 and use workspace package and dependencies ( #782 )
...
* chore: upgrade to Arrow 29.0 and use workspace package and dependencies
* fix: resolve PR comments
Co-authored-by: luofucong <luofucong@greptime.com >
2022-12-23 14:28:37 +08:00
Ruihang Xia
7ba512980a
chore: add APACHE-2.0 license header ( #518 )
...
* feat: add license checker workflow
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix existing header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* specify license for internal sub-crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 18:05:46 +08:00
Ruihang Xia
1565c8d236
chore: specify import style in rustfmt ( #460 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 15:58:54 +08:00
dennis zhuang
c8cb705d9e
ci: pre-commit configuration and hooks ( #261 )
...
* feat: adds pre-commit config and hooks
* refactor: sort all Cargo.toml by cargo-sort
* ci: adds conventional-pre-commit hook to pre-commit
* fix: remove .pre-commit-hooks.yaml
* fix: readme
* Update .pre-commit-config.yaml
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
* ci: move clippy hook to push stage
* docs: install pre-push github hook
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
2022-09-15 11:30:08 +08:00
LFC
ccda17248e
feat: unify servers and mysql server in datanode ( #172 )
...
* address PR comments
address PR comments
use 3306 for mysql server's default port
upgrade metric to version 0.20
move crate "servers" out of "common"
make mysql io threads count configurable in config file
add snafu backtrace for errors with source
use common-server error for mysql server
add test for grpc server
refactor testing codes
fix rustfmt check
start mysql server in datanode
move grpc server codes from datanode to common-servers
feat: unify servers
* rebase develop and resolve conflicts
* remove an unnecessary todo
Co-authored-by: luofucong <luofucong@greptime.com >
2022-08-17 14:29:12 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
dennis zhuang
379d2e2f50
feature: runtime crate and global runtimes ( #49 )
...
* feat: init common runtime crate
* feat: tokio Runtime wrapper and global runtime functions
* feat: adds block_on_read, block_on_write, block_on_bg functions to runtime
* refactor: panic when configure global runtimes which are already initialized
* refactor: adds read/write/bg thread pool size
* fix: code style
* fix: clippy warning
* fix: test_metric panic
* fix: address CR problems
* log: adds log when creating runtime
2022-06-21 16:09:15 +08:00
