Weny Xu
e0a43f37d7
chore: bump opendal to 0.44 ( #3058 )
...
* chore: bump opendal to 0.44
* fix: fix test_object_store_cache_policy
* Revert "fix: fix test_object_store_cache_policy"
This reverts commit 46c37c343f66114e0f6ee7a0a3b9ee2b79c810af.
* fix: fix test_object_store_cache_policy
* fix: fix test_file_backend_with_lru_cache
* chore: apply suggestions from CR
* fix(mito): fix mito2 cache
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-01-05 09:05:41 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
Weny Xu
93f3048f4f
refactor: migrate OpenDal to 0.39 ( #2383 )
...
* chore: bump opendal to 7d552
* refactor: migrate OpenDal to 0.39
* chore: apply suggestions from CR
2023-09-13 12:43:53 +00:00
Niwaka
66e5ed5483
feat: support gcs storage ( #1781 )
2023-07-05 23:03:51 +08:00
Lei, HUANG
f287d3115b
chore: replace result assertions ( #1840 )
...
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g
* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
2023-06-27 19:14:48 +08:00
LFC
fcff66e039
chore: deny unused results ( #1825 )
...
* chore: deny unused results
* rebase
2023-06-27 15:33:53 +08:00
dennis zhuang
f08f726bec
test: s3 manifest ( #1755 )
...
* feat: change default manifest options
* test: s3 manifest
* feat: revert checkpoint_margin to 10
* Update src/object-store/src/test_util.rs
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
---------
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com >
2023-06-09 10:28:41 +00:00
Zou Wei
ae81c7329d
feat: support azblob storage. ( #1659 )
...
* feat:support azblob storage.
* test:add some tests.
* refactor:use if-let.
2023-05-30 19:59:38 +08:00
dennis zhuang
7c55783e53
feat!: reorganize the storage layout ( #1609 )
...
* feat: adds data_home to DataOptions
* refactor: split out object store stuffs from datanode instance
* feat: move data_home into FileConfig
* refactor: object storage layers
* feat: adds datanode path to procedure paths
* feat: temp commit
* refactor: clean code
* fix: forgot files
* fix: forgot files
* Update src/common/test-util/src/ports.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update tests/runner/src/env.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* fix: compile error
* chore: cr comments
* fix: dependencies order in cargo
* fix: data path in test
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-05-23 13:58:26 +08:00
dennis zhuang
b1920c41a4
fix: object store cache bug ( #1482 )
...
* feat: use streaming read instead of reading whole file
* feat: enable atomic writing for object store file caching
* fix: recover existing keys from local cache
* test: recovering keys from local file cache for LruCachePolicy
* Update src/datanode/src/instance.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr comments
* feat: md5 hash caching path
* fix: test
* fix: read cache
* Update src/object-store/src/cache_policy.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-05-04 18:25:40 +08:00
Weny Xu
a77a4a4bd1
fix: add s3 region info ( #1492 )
...
fix: add region info
2023-04-27 19:13:01 +08:00
Near
c6f024a171
feat: Add metrics for cache hit/miss for object store cache ( #1405 )
...
* Add the cache hit/miss counter
* Verify the cache metrics are included
* Resolve comments
* Rename the error kind label name to be consistent with other metrics
* Rename the object store metric names
* Avoid using glob imports
* Format the code
* chore: Update src/object-store/src/metrics.rs mod doc
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-04-18 14:08:19 +08:00
Lei, HUANG
5edd2a3dbe
feat: upgrade opendal ( #1245 )
...
* chore: upgrade opendal
* chore: finish upgrading opendal
* fix: clippy complaints
* fix some tests
* fix: all unit tests
* chore: rebase develop
* fix: sqlness tests
* optimize imports
* chore: rebase develop
* doc: add todo
2023-03-28 09:47:33 +08:00
Eugene Tolbakov
b31a6cb506
refactor: replace tempdir with tempfile ( #1123 )
...
* refactor: replace tempdir with tempfile
* refactor(query): move tempfile dependency under the workspace's Cargo.toml
* refactor(tempfile): create common-test-util
* refactor(tempfile): fix toml format
* refactor(tempfile): remove tempfile out of dependencies
* refactor(tempfile): fix incorrect toml
2023-03-08 11:15:56 +08:00
Xuanwo
fdd17c6eeb
refactor: Clean up re-export of opendal services ( #1067 )
...
Signed-off-by: Xuanwo <github@xuanwo.io >
2023-02-23 14:12:34 +08:00
Xuanwo
98ef74bff4
chore: Bump OpenDAL to v0.27 ( #1057 )
...
* Bump OpenDAL to v0.27
Signed-off-by: Xuanwo <github@xuanwo.io >
* Make cargo check happy
Signed-off-by: Xuanwo <github@xuanwo.io >
* Address comments
Signed-off-by: Xuanwo <github@xuanwo.io >
* Address comments
Signed-off-by: Xuanwo <github@xuanwo.io >
* Format toml
Signed-off-by: Xuanwo <github@xuanwo.io >
* Make taplo happy
Signed-off-by: Xuanwo <github@xuanwo.io >
---------
Signed-off-by: Xuanwo <github@xuanwo.io >
2023-02-23 11:20:45 +08:00
elijah
926022e14c
feat: enable caching when using object store ( #928 )
...
* feat: enable caching when using object store
* feat: support file cache for object store
* feat: maintaining the cached files with lru
* fix: improve the code
* empty commit
* improve the code
2023-02-07 15:46:37 +08:00
Yun Chen
a7dc86ffe5
feat: oss storage support ( #911 )
...
* feat: add oss storage support
* fix: ci build format check
* fix: align OSS to Oss
* fix: cr comments
* fix: rename OSS to Oss in integration tests
* fix: clippy fix
2023-01-29 20:09:38 +08:00
Lei, HUANG
8ffc078f88
fix: license header ( #815 )
2023-01-03 15:09:49 +08:00
dennis zhuang
2e17e9c4b5
feat: supports s3 storage ( #656 )
...
* feat: adds s3 object storage configuration
* feat: adds s3 integration test
* chore: use map
* fix: forgot license header
* fix: checking if bucket is empty in test_on
* chore: address CR issues
* refactor: run s3 test with dotenv
* chore: randomize grpc port for test
* fix: README in tests-integration
* chore: remove redundant comments
2022-12-01 10:59:14 +08:00
dennis zhuang
5fddb799f7
feat: enable atomic write for file object storage ( #643 )
...
* fix: remove opendal from catalog dependencies
* feat: enable atomic writing for fs service
2022-11-28 16:01:32 +08:00
Xuanwo
4085fc7899
chore: Bump OpenDAL to v0.21.1 ( #639 )
...
* deps: Bump OpenDAL to v0.21.1
Signed-off-by: Xuanwo <github@xuanwo.io >
* Avoid using raw types when not needed
Signed-off-by: Xuanwo <github@xuanwo.io >
Signed-off-by: Xuanwo <github@xuanwo.io >
2022-11-27 10:18:39 +08:00
Xuanwo
1f0b39cc8d
chore: Bump OpenDAL to v0.20 ( #569 )
...
Signed-off-by: Xuanwo <github@xuanwo.io >
2022-11-18 14:17:38 +08:00
Ning Sun
173a8f67a1
test: ignore empty s3 bucket ( #529 )
2022-11-16 11:35:12 +08:00
Ruihang Xia
7ba512980a
chore: add APACHE-2.0 license header ( #518 )
...
* feat: add license checker workflow
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix existing header
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* specify license for internal sub-crate
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix rustfmt
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 18:05:46 +08:00
Ruihang Xia
1565c8d236
chore: specify import style in rustfmt ( #460 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2022-11-15 15:58:54 +08:00
Ning Sun
f243649971
refactor: Removed openssl from build requirement ( #308 )
...
* refactor:replace another axum-test-helper branch
* refactor: upgrade opendal version
* refactor: use cursor for file buffer
* refactor:remove native-tls in mysql_async
* refactor: use async block and pipeline for newer opendal api
* chore: update Cargo.lock
* chore: update dependencies
* docs: removed openssl from build requirement
* fix: call close on pipe writer to flush reader for parquet streamer
* refactor: remove redundant return
* chore: use pinned revision for our forked mysql_async
* style: avoid wild-card import in test code
* Apply suggestions from code review
Co-authored-by: Yingwen <realevenyag@gmail.com >
* style: use chained call for builder
Co-authored-by: liangxingjian <965662709@qq.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2022-10-17 19:29:17 +08:00
evenyag
bf5975ca3e
feat: Prototype of the storage engine ( #107 )
...
* feat: memtable flush (#63 )
* wip: memtable flush
* optimize schema conversion
* remove unnecessary import
* add parquet file verfication
* add backtrace to error
* chore: upgrade opendal to 0.9 and fixed some problems
* rename error
* fix: error description
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
* feat: region manifest service (#57 )
* feat: adds Manifest API
* feat: impl region manifest service
* refactor: by CR comments
* fix: storage error mod test
* fix: tweak storage cargo
* fix: tweak storage cargo
* refactor: by CR comments
* refactor: rename current_version
* feat: add wal writer (#60 )
* feat: add Wal
* upgrade engine for wal
* fix: unit test for wal
* feat: wal into region
* fix: unix test
* fix clippy
* chore: by cr
* chore: by cr
* chore: prevent test data polution
* chore: by cr
* minor fix
* chore: by cr
* feat: Implement flush (#65 )
* feat: Flush framework
- feat: Add id to memtable
- refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet
- feat: Freeze memtable
- feat: Trigger flush
- feat: Background job pool
- feat: flush job
- feat: Sst access layer
- feat: Custom Deserialize for StringBytes
- feat: Use RegionWriter to apply file metas
- feat: Apply version edit
- chore: Remove unused imports
refactor: Use ParquetWriter to replace FlushTask
refactor: FsAccessLayer takes object store as param
chore: Remove todo from doc comments
feat: Move wal to WriterContext
chore: Fix clippy
chore: Add backtrace to WriteWal error
* feat: adds manifest to region and refactor sst/manifest dir config (#72 )
* feat: adds manifest to region and refactor sst/manifest dir with EngineConfig
* refactor: ensure path ends with '/' in ManifestLogStorage
* fix: style
* refactor: normalize storage directory path and minor changes by CR
* refactor: doesn't need slash any more
* feat: Implement apply_edit() and add timestamp index to schema (#73 )
* feat: Implement VersionControl::apply_edit()
* feat: Add timestamp index to schema
* feat: Implement Schema::timestamp_column()
* feat: persist region metadata to manifest (#74 )
* feat: persist metadata when creating region or sst files
* fix: revert FileMeta comment
* feat: resolve todo
* fix: clippy warning
* fix: revert files_to_remove type in RegionEdit
* feat: impl SizeBasedStrategy for flush (#76 )
* feat: impl SizeBasedStrategy for flush
* doc: get_mutable_limitation
* fix: code style and comment
* feat: align timestamp (#75 )
* feat: align timestamps in write batch
* fix cr comments
* fix timestamp overflow
* simplify overflow check
* fix cr comments
* fix clippy issues
* test: Fix region tests (comment out some unsupported tests) (#82 )
* feat: flush job (#80 )
* feat: flush job
* fix cr comments
* move file name instead of clone
* comment log file test (#84 )
* feat: improve MemtableVersion (#78 )
* feat: improve MemtableVersion
* feat: remove flushed immutable memtables and test MemtableVersion
* refactor: by CR comments
* refactor: clone kv in iterator
* fix: clippy warning
* refactor: Make BatchIterator supertrait of Iterator (#85 )
* refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83 )
* feat: Insert multiple memtables by time range (#77 )
* feat: memtable::Inserter supports insert multiple memtables by time range
* chore: Update timestamp comment
* test: Add tests for Inserter
* test: Fix region tests (comment out some unsupported tests)
* refactor: align_timestamp() use TimestampMillis::aligned_by_bucket()
* chore: rename aligned_by_bucket to align_by_bucket
* fix: Fix compile errors
* fix: sst and manifest dir (#86 )
* Set RowKeyDescriptor::enable_version_column to false by default
* feat: Implement write stall (#90 )
* feat: Implement write stall
* chore: Update comments
* feat: Support reading multiple memtables (#93 )
* feat: Support reading multiple memtables
* test: uncomment tests rely on snapshot read
* feat: wal format (#70 )
* feat: wal codec
* chore: minor fix
* chore: comment
* chore: by cr
* chore: write_batch_codec mod
* chore: by cr
* chore: upgrade proto
* chore: by cr
* fix failing test
* fix failing test
* feat: manifest to wal (#100 )
* feat: write manifest to wal
* chore: sequence into wal
* chore: by cr
* chore: by cr
* refactor: create log store (#104 )
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
* chore: Fix clippy
Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com >
Co-authored-by: Dennis Zhuang <killme2008@gmail.com >
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me >
Co-authored-by: fariygirl <clickmetoday@163.com >
Co-authored-by: Lei, HUANG <mrsatangel@gmail.com >
2022-07-25 15:26:00 +08:00
dennis zhuang
a819b6bce5
feat: impl object-store based on opendal ( #27 )
...
* feat: impl object-store based on opendal
* feat: adds s3 test
* feat: export Layer
* feat: pass s3 access key/secret by env vars
* fix: logging when s3 test is running
2022-05-17 15:58:27 +08:00
