greptimedb

mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-25 05:12:27 +00:00

Author	SHA1	Message	Date
jeremyhi	bf67246645	perf: bound concurrent Metric export writers (#9296 ) * fix: preserve objects after conditional Metric export collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: share Metric export writer and payload budgets Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover bounded Metric export conversion and mixed roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve export output after ambiguous close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: align HTTP export timeout with managed cancellation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: trim redundant Metric export roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: bound pending Metric export writer handles Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: protect export files on failed writes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve ordinary export cleanup and large rows Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: report Metric export collisions as invalid arguments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed conditional export closes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support view arrays in managed ordinary export Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: expect successful cleanup after failed close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up unsynced overwrite after close failure Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: scope export JSON expansion estimates to JSON columns Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: rechunk ordinary export batches within the write budget Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2026-09-28 03:10:21 +00:00
dennis zhuang	f3eb8e6c72	test: cut integration test time and make the storage matrix meaningful (#9308 ) * test: cut integration test time and make the storage matrix meaningful tests-integration is ~85% of workspace test CPU, and 81% of that is the S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not touch the object store: of the 70 matrix HTTP tests only one flushed and read back an SST, so the matrix was paying real AWS round trips to re-prove protocol parsing. - Point the PR CI object-store matrix at the MinIO already started by tests-integration/fixtures. Three GT_S3_* consumers did not read GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials; they now do. - Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the release integration-test container. The release previously ran every remote-backend case as a skip and only exercised the file backend. - Give each S3WithCache test its own read cache directory. They shared /tmp/greptimedb_cache, which the datanode wipes on startup, so a starting test deleted the read cache of a running one. - Add flush -> read-back assertions to the tests whose columns have a non-trivial SST representation: JSON/JSON2 columns, native histograms, metric-engine logical tables, and tables carrying fulltext or skipping indexes whose puffin files only exist after a flush. - Move eight tests that create no table out of the storage matrix. - Make the event recorder flush interval a constructor parameter and shorten it in the event tests, which otherwise wait a 5s window per DDL they assert on. It is skipped by serde and never reaches config files. - Drop duplicates: test_grpc_zstd_compression was a verbatim copy of test_grpc_message_size_ok and is now rewritten to assert the negotiated grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict prefixes of their copy_from siblings; two standalone/distributed event test pairs shared one assertion body. - Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates in a parallelism-dependent order, so its last digits are unstable; the test now compares values with a tolerance. - Rebase the jaeger v1 fixture on the current instant. It carries ttl=7d with 2025 timestamps, so its rows were only readable as long as they stayed in the memtable. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: address review — wire nightly real-S3 job into check-status, keep the short event interval The nightly `check-status` job did not depend on the new real-S3 job, so a failure there would not have reached the status or Slack notification. In database_ddl_event the short interval was set by a first `with_event_recorder_options` call and then overwritten by the pre-existing one, which carries `..Default::default()`. Merged into a single call. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-09-23 01:52:52 +00:00
XuanwoandWenyXu	4c12ea1aba	chore(deps): bump opendal to 0.58.1 (#8742 ) * chore(deps): bump opendal to 0.58.1 Upgrade direct opendal dependency and workspace object_store_opendal pin from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal 0.58.0). Adapt to OpenDAL 0.58 composition API: - Operator::new returns a finished operator; drop .finish() call sites - Replace HttpClientLayer / raw::HttpClient with OperationContext + HttpTransporter (ReqwestTransport) - Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to Service + Layer::apply_service - Rewrite SecureFs reader/writer/lister for sync factories and StreamRead - Use OperatorInfo::capability() instead of removed native_capability() Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: retrigger CI after udeps runner segfault Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(object-store): restore suffix read simulation for secure fs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: adapt remaining callers to OpenDAL 0.58 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com>	2026-09-07 08:16:47 +00:00
Ruihang Xiaanddiscord9	3b3f5d628d	feat: bump datafusion to 53 (#8107 ) * feat: bump datafusion to 53 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * use updated datafusion-orc Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * maintain topk in part sort Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * docs: clarify part_sort range detection and Null threshold semantics * fix limit 0 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: discord9 <discord9@163.com>	2026-05-14 08:33:31 +00:00
shuiyisongandevenyag	0effc30778	chore: update the opendal to 0.56 rc2 (#8003 ) * chore: update opendal version Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update opendal version Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: grpc init Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: dep versions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove aws-lc-rs in reqwest Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rebase main and fix compile Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove unused deps Signed-off-by: shuiyisong <xixing.sys@gmail.com> * Revert "fix: remove aws-lc-rs in reqwest" This reverts commit 90bfafca06f877befb36f3e54bb72fcfc4c56778. * chore: remove aws-lc-sys from blacklist Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tls deps Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: idemptent install in rds Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use aws-lc-sys as possible Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: lint Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: address comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: address CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sync opendal compat adapter with upstream Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address compat clippy warnings Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: evenyag <realevenyag@gmail.com>	2026-04-26 09:59:48 +00:00
Yingwen	dc2f2cbfae	fix: manifest recovery scans after last version if possible (#8009 ) * feat: suppport scan with start after Signed-off-by: evenyag <realevenyag@gmail.com> * test: add start_after test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: adjust remove dir warning Signed-off-by: evenyag <realevenyag@gmail.com> * test: test list_with_start_after Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update get_paths call with start_after arg in checkpoint test Signed-off-by: evenyag <realevenyag@gmail.com> * feat: log scan metrics Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix start_after on manifest dir Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2026-04-23 02:42:59 +00:00
Yingwen	39d3744f4f	fix: bump prometheus to 0.14 (#7686 ) * chore: add metrics test for fs backend Signed-off-by: evenyag <realevenyag@gmail.com> * test: test opendal metrics Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update prometheus to 0.14 The opendal metrics require 0.14. Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2026-02-09 09:51:00 +00:00
Yingwen	f6afb10e33	feat!: download file to fill the cache on write cache miss (#7294 ) * feat: download inverted index file Signed-off-by: evenyag <realevenyag@gmail.com> * feat: download for bloom and fulltext Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement maybe_download_background for FileCache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: load file for parquet Signed-off-by: evenyag <realevenyag@gmail.com> * chore: reduce channel size Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use ManifestCache Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass cache to ManifestObjectStore::new Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix fmt and clippy Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove manifest cache ttl Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove read cache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clean old read cache path Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config examples Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix CI Signed-off-by: evenyag <realevenyag@gmail.com> * fix: also clean the root directory Signed-off-by: evenyag <realevenyag@gmail.com> * test: update manifest test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * feat: skip file if it exists Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove warn in replace Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add a flag to enable/disable background download set the concurrency to 1 for background download Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename write_cache_enable_background_download to enable_refill_cache_on_read Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address comments Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config.md Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-16 08:31:26 +00:00
Ruihang Xia	c9377e7c5a	build: bump rust edition to 2024 (#6920 ) * bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-09-08 02:37:18 +00:00
Ruihang Xia	a23f269bb1	fix: correct write cache's metric labels (#5227 ) * refactor: remove unused field in WriteCache Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor: unify read and write cache path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update config and fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove unnecessary methods and adapt test Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change the default path Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove remote-home Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2024-12-25 07:26:21 +00:00
Ruihang Xia	422d18da8b	feat: bump opendal and switch prometheus layer to the upstream impl (#5179 ) * feat: bump opendal and switch prometheus layer to the upstream impl Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove unused files Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove unused things Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove root dir on recovering cache Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * filter out non-files entry in test Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2024-12-19 03:42:05 +00:00
Weny Xu	51c6eafb16	feat: recover file cache index asynchronously (#5087 ) * feat: recover file cache index asynchronously * chore: apply suggestions from CR	2024-12-03 09:33:52 +00:00
liyang	f578155602	feat: add GcsConfig credential field (#4568 )	2024-08-16 03:11:20 +00:00
Weny Xu	cb4cffe636	chore: bump opendal version to 0.48 (#4499 )	2024-08-04 00:46:04 +00:00
tisonandXuanwo	2665616f72	build(deps): Upgrade OpenDAL to 0.47 (#4224 ) * catch up changes Signed-off-by: tison <wander4096@gmail.com> * fmt Signed-off-by: tison <wander4096@gmail.com> * Fix cache for 0471 (#7) * Fix cache for 0471 Signed-off-by: Xuanwo <github@xuanwo.io> * Make clippy happy Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io> * tidy Signed-off-by: tison <wander4096@gmail.com> * use opendal's exported type Signed-off-by: tison <wander4096@gmail.com> * clippy Signed-off-by: tison <wander4096@gmail.com> * fmt Signed-off-by: tison <wander4096@gmail.com> --------- Signed-off-by: tison <wander4096@gmail.com> Signed-off-by: Xuanwo <github@xuanwo.io> Co-authored-by: Xuanwo <github@xuanwo.io>	2024-07-01 17:05:15 +00:00
tisonandXuanwo	f9db5ff0d6	build(deps): upgrade opendal to 0.46 (#4037 ) * build(deps): upgrade opendal to 0.46 Signed-off-by: tison <wander4096@gmail.com> * migrate writes Signed-off-by: tison <wander4096@gmail.com> * migrate reads Signed-off-by: tison <wander4096@gmail.com> * fixup object safety Signed-off-by: tison <wander4096@gmail.com> * fixup names Signed-off-by: tison <wander4096@gmail.com> * fixup compilation Signed-off-by: tison <wander4096@gmail.com> * fixup compilation Signed-off-by: tison <wander4096@gmail.com> * a few Buffer to Vec Signed-off-by: tison <wander4096@gmail.com> * Make greptime buildable with opendal 0.46 (#5) Signed-off-by: Xuanwo <github@xuanwo.io> * fixup toml check Signed-off-by: tison <wander4096@gmail.com> * test_orc_opener Signed-off-by: tison <wander4096@gmail.com> * Fix lru cache (#6) Signed-off-by: Xuanwo <github@xuanwo.io> * clippy Signed-off-by: tison <wander4096@gmail.com> * improve comments Signed-off-by: tison <wander4096@gmail.com> * address comments Signed-off-by: tison <wander4096@gmail.com> * reduce buf copy Signed-off-by: tison <wander4096@gmail.com> * upgrade to reqwest 0.12 Signed-off-by: tison <wander4096@gmail.com> --------- Signed-off-by: tison <wander4096@gmail.com> Signed-off-by: Xuanwo <github@xuanwo.io> Co-authored-by: Xuanwo <github@xuanwo.io>	2024-05-27 09:12:23 +00:00
Ruihang Xia	530353785c	refactor: remove re-export from logging (#3865 ) * refactor: remove re-export from logging Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix merge problem Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * run formatter Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2024-05-06 13:26:01 +00:00
Weny Xu	e0a43f37d7	chore: bump opendal to 0.44 (#3058 ) * chore: bump opendal to 0.44 * fix: fix test_object_store_cache_policy * Revert "fix: fix test_object_store_cache_policy" This reverts commit 46c37c343f66114e0f6ee7a0a3b9ee2b79c810af. * fix: fix test_object_store_cache_policy * fix: fix test_file_backend_with_lru_cache * chore: apply suggestions from CR * fix(mito): fix mito2 cache * chore: apply suggestions from CR * chore: apply suggestions from CR	2024-01-05 09:05:41 +00:00
Ruihang Xia	6b8dbcfb54	chore: update toolchain to 20231219 (#2932 ) * update toolchain file, remove unused feature gates Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update action file Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update to 12-19 Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2023-12-19 07:24:08 +00:00
WU Jingdi	180bc64cb0	feat: change metric crate from metrics to prometheus (#2655 ) * feat: change metrics to prometheus * chore: fix code advise * chore: resolve merge conflict * chore: fix code advise	2023-10-31 15:46:57 +00:00
dennis zhuang Ruihang Xia Yingwen	ff15bc41d6	feat: improve object storage cache (#2522 ) * feat: refactor object storage cache with moka * chore: minor fixes * fix: concurrent issues and invalidate cache after write/delete * chore: minor changes * fix: cargo lock * refactor: rename * chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib * fix: typo * chore: style * fix: toml format * chore: toml * fix: toml format * Update src/object-store/src/layers/lru_cache/read_cache.rs Co-authored-by: Ruihang Xia <waynestxia@gmail.com> * chore: update Cargo.toml Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: update src/object-store/Cargo.toml Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: refactor and apply suggestions * fix: typo * feat: adds back allow list for caching * chore: cr suggestion Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: cr suggestion Co-authored-by: Yingwen <realevenyag@gmail.com> * refactor: wrap inner Accessor with Arc * chore: remove run_pending_task in read and write path * chore: the arc is unnecessary --------- Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com>	2023-10-08 03:27:49 +00:00
Weny Xu	93f3048f4f	refactor: migrate OpenDal to 0.39 (#2383 ) * chore: bump opendal to 7d552 * refactor: migrate OpenDal to 0.39 * chore: apply suggestions from CR	2023-09-13 12:43:53 +00:00
Niwaka	66e5ed5483	feat: support gcs storage (#1781 )	2023-07-05 23:03:51 +08:00
Lei, HUANG	f287d3115b	chore: replace result assertions (#1840 ) * s/assert!\((.)\.is_ok\(\)\);/\1.unwrap\(\);/g s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g	2023-06-27 19:14:48 +08:00
LFC	fcff66e039	chore: deny unused results (#1825 ) * chore: deny unused results * rebase	2023-06-27 15:33:53 +08:00
dennis zhuangandLei, HUANG	f08f726bec	test: s3 manifest (#1755 ) * feat: change default manifest options * test: s3 manifest * feat: revert checkpoint_margin to 10 * Update src/object-store/src/test_util.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> --------- Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>	2023-06-09 10:28:41 +00:00
Zou Wei	ae81c7329d	feat: support azblob storage. (#1659 ) * feat:support azblob storage. * test:add some tests. * refactor:use if-let.	2023-05-30 19:59:38 +08:00
dennis zhuangandYingwen	7c55783e53	feat!: reorganize the storage layout (#1609 ) * feat: adds data_home to DataOptions * refactor: split out object store stuffs from datanode instance * feat: move data_home into FileConfig * refactor: object storage layers * feat: adds datanode path to procedure paths * feat: temp commit * refactor: clean code * fix: forgot files * fix: forgot files * Update src/common/test-util/src/ports.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * Update tests/runner/src/env.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * fix: compile error * chore: cr comments * fix: dependencies order in cargo * fix: data path in test --------- Co-authored-by: Yingwen <realevenyag@gmail.com>	2023-05-23 13:58:26 +08:00
dennis zhuangandYingwen	b1920c41a4	fix: object store cache bug (#1482 ) * feat: use streaming read instead of reading whole file * feat: enable atomic writing for object store file caching * fix: recover existing keys from local cache * test: recovering keys from local file cache for LruCachePolicy * Update src/datanode/src/instance.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * chore: cr comments * feat: md5 hash caching path * fix: test * fix: read cache * Update src/object-store/src/cache_policy.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Co-authored-by: Yingwen <realevenyag@gmail.com>	2023-05-04 18:25:40 +08:00
Weny Xu	a77a4a4bd1	fix: add s3 region info (#1492 ) fix: add region info	2023-04-27 19:13:01 +08:00
NearandYingwen	c6f024a171	feat: Add metrics for cache hit/miss for object store cache (#1405 ) * Add the cache hit/miss counter * Verify the cache metrics are included * Resolve comments * Rename the error kind label name to be consistent with other metrics * Rename the object store metric names * Avoid using glob imports * Format the code * chore: Update src/object-store/src/metrics.rs mod doc --------- Co-authored-by: Yingwen <realevenyag@gmail.com>	2023-04-18 14:08:19 +08:00
Lei, HUANG	5edd2a3dbe	feat: upgrade opendal (#1245 ) * chore: upgrade opendal * chore: finish upgrading opendal * fix: clippy complaints * fix some tests * fix: all unit tests * chore: rebase develop * fix: sqlness tests * optimize imports * chore: rebase develop * doc: add todo	2023-03-28 09:47:33 +08:00
Eugene Tolbakov	b31a6cb506	refactor: replace tempdir with tempfile (#1123 ) * refactor: replace tempdir with tempfile * refactor(query): move tempfile dependency under the workspace's Cargo.toml * refactor(tempfile): create common-test-util * refactor(tempfile): fix toml format * refactor(tempfile): remove tempfile out of dependencies * refactor(tempfile): fix incorrect toml	2023-03-08 11:15:56 +08:00
Xuanwo	fdd17c6eeb	refactor: Clean up re-export of opendal services (#1067 ) Signed-off-by: Xuanwo <github@xuanwo.io>	2023-02-23 14:12:34 +08:00
Xuanwo	98ef74bff4	chore: Bump OpenDAL to v0.27 (#1057 ) * Bump OpenDAL to v0.27 Signed-off-by: Xuanwo <github@xuanwo.io> * Make cargo check happy Signed-off-by: Xuanwo <github@xuanwo.io> * Address comments Signed-off-by: Xuanwo <github@xuanwo.io> * Address comments Signed-off-by: Xuanwo <github@xuanwo.io> * Format toml Signed-off-by: Xuanwo <github@xuanwo.io> * Make taplo happy Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>	2023-02-23 11:20:45 +08:00
elijah	926022e14c	feat: enable caching when using object store (#928 ) * feat: enable caching when using object store * feat: support file cache for object store * feat: maintaining the cached files with lru * fix: improve the code * empty commit * improve the code	2023-02-07 15:46:37 +08:00
Yun Chen	a7dc86ffe5	feat: oss storage support (#911 ) * feat: add oss storage support * fix: ci build format check * fix: align OSS to Oss * fix: cr comments * fix: rename OSS to Oss in integration tests * fix: clippy fix	2023-01-29 20:09:38 +08:00
Lei, HUANG	8ffc078f88	fix: license header (#815 )	2023-01-03 15:09:49 +08:00
dennis zhuang	2e17e9c4b5	feat: supports s3 storage (#656 ) * feat: adds s3 object storage configuration * feat: adds s3 integration test * chore: use map * fix: forgot license header * fix: checking if bucket is empty in test_on * chore: address CR issues * refactor: run s3 test with dotenv * chore: randomize grpc port for test * fix: README in tests-integration * chore: remove redundant comments	2022-12-01 10:59:14 +08:00
dennis zhuang	5fddb799f7	feat: enable atomic write for file object storage (#643 ) * fix: remove opendal from catalog dependencies * feat: enable atomic writing for fs service	2022-11-28 16:01:32 +08:00
Xuanwo	4085fc7899	chore: Bump OpenDAL to v0.21.1 (#639 ) * deps: Bump OpenDAL to v0.21.1 Signed-off-by: Xuanwo <github@xuanwo.io> * Avoid using raw types when not needed Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: Xuanwo <github@xuanwo.io>	2022-11-27 10:18:39 +08:00
Xuanwo	1f0b39cc8d	chore: Bump OpenDAL to v0.20 (#569 ) Signed-off-by: Xuanwo <github@xuanwo.io>	2022-11-18 14:17:38 +08:00
Ning Sun	173a8f67a1	test: ignore empty s3 bucket (#529 )	2022-11-16 11:35:12 +08:00
Ruihang Xia	7ba512980a	chore: add APACHE-2.0 license header (#518 ) * feat: add license checker workflow Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix existing header Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * specify license for internal sub-crate Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix rustfmt Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2022-11-15 18:05:46 +08:00
Ruihang Xia	1565c8d236	chore: specify import style in rustfmt (#460 ) Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2022-11-15 15:58:54 +08:00
Ning Sun Yingwen liangxingjian	f243649971	refactor: Removed openssl from build requirement (#308 ) * refactor:replace another axum-test-helper branch * refactor: upgrade opendal version * refactor: use cursor for file buffer * refactor:remove native-tls in mysql_async * refactor: use async block and pipeline for newer opendal api * chore: update Cargo.lock * chore: update dependencies * docs: removed openssl from build requirement * fix: call close on pipe writer to flush reader for parquet streamer * refactor: remove redundant return * chore: use pinned revision for our forked mysql_async * style: avoid wild-card import in test code * Apply suggestions from code review Co-authored-by: Yingwen <realevenyag@gmail.com> * style: use chained call for builder Co-authored-by: liangxingjian <965662709@qq.com> Co-authored-by: Yingwen <realevenyag@gmail.com>	2022-10-17 19:29:17 +08:00
evenyag dennis zhuang Lei, Huang fariygirl Jiachun Feng Lei, HUANG	bf5975ca3e	feat: Prototype of the storage engine (#107 ) * feat: memtable flush (#63) * wip: memtable flush * optimize schema conversion * remove unnecessary import * add parquet file verfication * add backtrace to error * chore: upgrade opendal to 0.9 and fixed some problems * rename error * fix: error description Co-authored-by: Dennis Zhuang <killme2008@gmail.com> * feat: region manifest service (#57) * feat: adds Manifest API * feat: impl region manifest service * refactor: by CR comments * fix: storage error mod test * fix: tweak storage cargo * fix: tweak storage cargo * refactor: by CR comments * refactor: rename current_version * feat: add wal writer (#60) * feat: add Wal * upgrade engine for wal * fix: unit test for wal * feat: wal into region * fix: unix test * fix clippy * chore: by cr * chore: by cr * chore: prevent test data polution * chore: by cr * minor fix * chore: by cr * feat: Implement flush (#65) * feat: Flush framework - feat: Add id to memtable - refactor: Rename MemtableSet/MutableMemtables to MemtableVersion/MemtableSet - feat: Freeze memtable - feat: Trigger flush - feat: Background job pool - feat: flush job - feat: Sst access layer - feat: Custom Deserialize for StringBytes - feat: Use RegionWriter to apply file metas - feat: Apply version edit - chore: Remove unused imports refactor: Use ParquetWriter to replace FlushTask refactor: FsAccessLayer takes object store as param chore: Remove todo from doc comments feat: Move wal to WriterContext chore: Fix clippy chore: Add backtrace to WriteWal error * feat: adds manifest to region and refactor sst/manifest dir config (#72) * feat: adds manifest to region and refactor sst/manifest dir with EngineConfig * refactor: ensure path ends with '/' in ManifestLogStorage * fix: style * refactor: normalize storage directory path and minor changes by CR * refactor: doesn't need slash any more * feat: Implement apply_edit() and add timestamp index to schema (#73) * feat: Implement VersionControl::apply_edit() * feat: Add timestamp index to schema * feat: Implement Schema::timestamp_column() * feat: persist region metadata to manifest (#74) * feat: persist metadata when creating region or sst files * fix: revert FileMeta comment * feat: resolve todo * fix: clippy warning * fix: revert files_to_remove type in RegionEdit * feat: impl SizeBasedStrategy for flush (#76) * feat: impl SizeBasedStrategy for flush * doc: get_mutable_limitation * fix: code style and comment * feat: align timestamp (#75) * feat: align timestamps in write batch * fix cr comments * fix timestamp overflow * simplify overflow check * fix cr comments * fix clippy issues * test: Fix region tests (comment out some unsupported tests) (#82) * feat: flush job (#80) * feat: flush job * fix cr comments * move file name instead of clone * comment log file test (#84) * feat: improve MemtableVersion (#78) * feat: improve MemtableVersion * feat: remove flushed immutable memtables and test MemtableVersion * refactor: by CR comments * refactor: clone kv in iterator * fix: clippy warning * refactor: Make BatchIterator supertrait of Iterator (#85) * refactor: rename Version to ManifestVersion and move out manifest from ShareData (#83) * feat: Insert multiple memtables by time range (#77) * feat: memtable::Inserter supports insert multiple memtables by time range * chore: Update timestamp comment * test: Add tests for Inserter * test: Fix region tests (comment out some unsupported tests) * refactor: align_timestamp() use TimestampMillis::aligned_by_bucket() * chore: rename aligned_by_bucket to align_by_bucket * fix: Fix compile errors * fix: sst and manifest dir (#86) * Set RowKeyDescriptor::enable_version_column to false by default * feat: Implement write stall (#90) * feat: Implement write stall * chore: Update comments * feat: Support reading multiple memtables (#93) * feat: Support reading multiple memtables * test: uncomment tests rely on snapshot read * feat: wal format (#70) * feat: wal codec * chore: minor fix * chore: comment * chore: by cr * chore: write_batch_codec mod * chore: by cr * chore: upgrade proto * chore: by cr * fix failing test * fix failing test * feat: manifest to wal (#100) * feat: write manifest to wal * chore: sequence into wal * chore: by cr * chore: by cr * refactor: create log store (#104) Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: fariygirl <clickmetoday@163.com> Co-authored-by: Jiachun Feng <jiachun_feng@proton.me> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com> * chore: Fix clippy Co-authored-by: Lei, Huang <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Jiachun Feng <jiachun_feng@proton.me> Co-authored-by: fariygirl <clickmetoday@163.com> Co-authored-by: Lei, HUANG <mrsatangel@gmail.com>	2022-07-25 15:26:00 +08:00
dennis zhuang	a819b6bce5	feat: impl object-store based on opendal (#27 ) * feat: impl object-store based on opendal * feat: adds s3 test * feat: export Layer * feat: pass s3 access key/secret by env vars * fix: logging when s3 test is running	2022-05-17 15:58:27 +08:00