greptimedb

mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-14 20:10:37 +00:00

Author	SHA1	Message	Date
Yingwen	f6afb10e33	feat!: download file to fill the cache on write cache miss (#7294 ) * feat: download inverted index file Signed-off-by: evenyag <realevenyag@gmail.com> * feat: download for bloom and fulltext Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement maybe_download_background for FileCache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: load file for parquet Signed-off-by: evenyag <realevenyag@gmail.com> * chore: reduce channel size Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use ManifestCache Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass cache to ManifestObjectStore::new Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix fmt and clippy Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove manifest cache ttl Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove read cache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clean old read cache path Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config examples Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix CI Signed-off-by: evenyag <realevenyag@gmail.com> * fix: also clean the root directory Signed-off-by: evenyag <realevenyag@gmail.com> * test: update manifest test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * feat: skip file if it exists Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove warn in replace Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add a flag to enable/disable background download set the concurrency to 1 for background download Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename write_cache_enable_background_download to enable_refill_cache_on_read Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address comments Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config.md Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-16 08:31:26 +00:00
Weny Xu	f7d5c87ac0	feat: introduce `copy_region_from` for mito engine (#7389 ) * feat: introduce `copy_region_from` Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-16 06:12:06 +00:00
LFC	f1abe5d215	feat: suspend frontend and datanode (#7370 ) Signed-off-by: luofucong <luofc@foxmail.com>	2025-12-10 12:18:24 +00:00
discord9	1ebcef4794	chore: rm unnecessary warning (#7352 ) Signed-off-by: discord9 <discord9@163.com>	2025-12-05 03:44:45 +00:00
Ruihang Xia	9d35b8cad4	refactor: remove datafusion data frame wrapper (#7347 ) Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-12-04 09:38:41 +00:00
Weny Xu	18875eed4d	feat: implement `Display` trait for `FlushRegions` (#7320 ) feat: implement Display trait for FlushRegions Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-01 06:33:23 +00:00
Weny Xu	8346acb900	feat: introduce `EnterStagingRequest` for `RegionEngine` (#7261 ) * feat: introduce `EnterStagingRequest` for region engine Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: improve error handling in staging mode entry Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-11-28 09:02:32 +00:00
Weny Xu	6b6d1ce7c4	feat: introduce `remap_manifests` for `RegionEngine` (#7265 ) * refactor: consolidate RegionManifestOptions creation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce`remap_manifests` for `RegionEngine` Signed-off-by: WenyXu <wenymedia@gmail.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-25 12:09:20 +00:00
discord9	0cee4fa115	feat: gc get ref from manifest (#7260 ) feat: get file ref from other manifest Signed-off-by: discord9 <discord9@163.com>	2025-11-19 12:13:28 +00:00
discord9	e59612043d	feat: gc scheduler ctx&procedure (#7252 ) * feat: gc ctx&procedure Signed-off-by: discord9 <discord9@163.com> * fix: handle region not found case Signed-off-by: discord9 <discord9@163.com> * docs: more explain&todo Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: add time for region gc Signed-off-by: discord9 <discord9@163.com> * fix: explain why loader for gc region should fail Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-19 08:35:17 +00:00
discord9	29bbff3c90	feat: gc worker only local regions&test (#7203 ) * feat: gc worker only on local region Signed-off-by: discord9 <discord9@163.com> * more check Signed-off-by: discord9 <discord9@163.com> * chore: stuff Signed-off-by: discord9 <discord9@163.com> * fix: ignore async index file for now Signed-off-by: discord9 <discord9@163.com> * fix: file removal rate calc Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-18 02:45:09 +00:00
WaterWhisperer	de9ae6066f	refactor: remove `export_metrics` and related configuration (#7236 ) Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>	2025-11-17 02:32:22 +00:00
jeremyhi	c7fded29ee	feat: query mem limiter (#7078 ) * feat: query mem limiter * feat: config docs * feat: frontend query limit config * fix: unused imports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add metrics for query memory tracker Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: right postion for tracker Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid race condition Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: soft and hard limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: when soft_limit == 0 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: upgrade limit algorithm Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: remove batch window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: batch mem size Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: refine limit algorithm Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: get sys mem Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change * feat: up tracker to the top stream * feat: estimated_size for batch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor refactor * feat: scan_memory_limit connect to max_concurrent_queries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make callback clearly * feat: add unlimted enum Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment * chore: comment on recursion_limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: refactor and put permit into RegionScanExec Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: multiple lazy static blocks * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2025-11-11 07:47:55 +00:00
Lei, HUANG	abcfbd7f41	chore(metrics): add region server requests failures count metrics (#7173 ) * chore/add-region-insert-failure-metric: Add metric for failed insert requests to region server in datanode module Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/add-region-insert-failure-metric: Add metric for tracking failed region server requests - Introduce a new metric `REGION_SERVER_REQUEST_FAILURE_COUNT` to count failed region server requests. - Update `REGION_SERVER_INSERT_FAIL_COUNT` metric description for consistency. - Implement error handling in `RegionServerHandler` to increment the new failure metric on request errors. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-11-05 07:23:40 +00:00
Weny Xu	9de680f456	refactor: add support for batch region upgrade operations part2 (#7160 ) * add tests for metric engines Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: catchup in background Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: replace sequential catchup with batch processing Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * remove single catchup Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused error Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine catchup tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-11-03 06:01:38 +00:00
Weny Xu	6960a0183a	refactor: add support for batch region upgrade operations part1 (#7155 ) * refactor: convert UpgradeRegion instruction to batch operation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce `handle_batch_catchup_requests` fn for mito engine Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce `handle_batch_catchup_requests` fn for metric engine Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: suggestion and add ser/de tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-10-31 03:08:38 +00:00
Lei, HUANG	f0afd675e3	feat: objbench sub command for datanode (#7114 ) * feat/objbench-subcmd: ### Add Object Storage Benchmark Tool and Update Dependencies - `Cargo.lock` & `Cargo.toml`: Added dependencies for `colored`, `parquet`, and `pprof` to support new features. - `datanode.rs`: Introduced `ObjbenchCommand` for benchmarking object storage, including command-line options for configuration and execution. Added `StorageConfig` and `StorageConfigWrapper` for storage engine configuration. - `datanode.rs`: Implemented a stub for `build_object_store` function to initialize object storage. These changes introduce a new subcommand for object storage benchmarking and update dependencies to support additional functionality. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * init Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: code style and clippy * feat/objbench-subcmd: Improve error handling in `objbench.rs` - Enhanced error handling in `parse_config` and `parse_file_dir_components` functions by replacing `unwrap` with `OptionExt` and `context` for better error messages. - Updated `build_access_layer_simple` and `build_cache_manager` functions to use `map_err` for more descriptive error handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: rebase main Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-10-29 05:26:29 +00:00
discord9	37bc2e6b07	feat: gc worker heartbeat instruction (#7118 ) again false by default test: config api refactor: per code review less info! even less info!! docs: gc regions instr refactor: grp by region id per code review per review error handling? test: fix todos aft rebase fix after refactor Signed-off-by: discord9 <discord9@163.com>	2025-10-29 02:59:36 +00:00
discord9	f2bc92b9e6	refactor: use generic for heartbeat instruction handler (#7149 ) * refactor: use generic Signed-off-by: discord9 <discord9@163.com> * w Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-10-27 09:09:48 +00:00
shuiyisong	a20ac4f9e5	feat: prefix option for timestamp index and value column (#7125 ) * refactor: use GREPTIME_TIMESTAMP const Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: add config for default ts col name Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: replace GREPTIME_TIMESTAMP with function get Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update config doc * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove opts on flownode and metasrv Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add validation for ts column name Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use get_or_init to avoid test error Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: using empty string to disable prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update comment Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: address CR issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>	2025-10-27 08:00:03 +00:00
Weny Xu	7da2f5ed12	refactor: refactor instruction handler and adds support for batch region downgrade operations (#7130 ) * refactor: refactor instruction handler Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: support batch downgrade region instructions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix compat Signed-off-by: WenyXu <wenymedia@gmail.com> * fix clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-10-24 09:11:42 +00:00
zyy17	a0e6bcbeb3	feat: add `cpu_usage_millicores` and `memory_usage_bytes` in `information_schema.cluster_info` table. (#7051 ) * refactor: add `hostname` in cluster_info table Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: update information schema result Signed-off-by: zyy17 <zyylsxm@gmail.com> * feat: enable zstd for bulk memtable encoded parts (#7045) feat: enable zstd in bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: add `get_total_cpu_millicores()` / `get_total_cpu_cores()` / `get_total_memory_bytes()` / `get_total_memory_readable()` in common-stat Signed-off-by: zyy17 <zyylsxm@gmail.com> * feat: add `cpu_usage_millicores` and `memory_usage_bytes` in `information_schema.cluster_info` table Signed-off-by: zyy17 <zyylsxm@gmail.com> * fix: compile warning and integration test failed Signed-off-by: zyy17 <zyylsxm@gmail.com> * fix: integration test failed Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: add `ResourceStat` Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: apply code review comments Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: update greptime-proto Signed-off-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: zyy17 <zyylsxm@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com>	2025-10-24 03:12:45 +00:00
Weny Xu	328ec56b63	feat: introduce `OpenRegions` and `CloseRegions` instructions to support batch region operations (#7122 ) * feat: introduce `OpenRegions` and `CloseRegions` instructions to support batch region operations Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: merge instructions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-10-22 03:43:47 +00:00
Sicong Hu	a1af4dce0c	feat: implement three build types for async index build (#7029 ) * feat: impl four types index build Signed-off-by: SNC123 <sinhco@outlook.com> * test: add tests for four types index build Signed-off-by: SNC123 <sinhco@outlook.com> * test: add sqlness test for manual index build Signed-off-by: SNC123 <sinhco@outlook.com> * fix: add region request support and correct sqlness Signed-off-by: SNC123 <sinhco@outlook.com> * fix: update cargo.toml for proto and resolve conflicts Signed-off-by: SNC123 <sinhco@outlook.com> * fix: rebase Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy Signed-off-by: SNC123 <sinhco@outlook.com> * fix: toml fmt and correct sqlness Signed-off-by: SNC123 <sinhco@outlook.com> * fix: correct sqlness result Signed-off-by: SNC123 <sinhco@outlook.com> * refactor: extract manual build logic Signed-off-by: SNC123 <sinhco@outlook.com> * apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * feat: abort index build process Signed-off-by: SNC123 <sinhco@outlook.com> * clippy Signed-off-by: SNC123 <sinhco@outlook.com> * chore: wrap `should_abort_index` Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com>	2025-10-21 02:48:28 +00:00
Zhenchi	938d757523	feat: expose SST index metadata via information schema (#7044 ) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-10-20 11:59:16 +00:00
Weny Xu	3119464ff9	feat: introduce the Noop WAL provider for datanode (#7105 ) * feat: introduce noop log store Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update config example Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add noop wal tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-10-20 06:13:27 +00:00
fys	20b5b9bee4	chore: remove unused deps (#7108 )	2025-10-17 11:53:19 +00:00
zyy17	6ab4672866	refactor: add `peer_hostname` field in `information_schema.cluster_info` table (#7050 ) * refactor: add `hostname` in cluster_info table Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: update information schema result Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: apply code review comments Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: update greptime-proto Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: add the compatibility for old proto Signed-off-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: zyy17 <zyylsxm@gmail.com>	2025-10-16 06:02:47 +00:00
LFC	8fe17d43d5	chore: update rust to nightly 2025-10-01 (#7069 ) * chore: update rust to nightly 2025-10-01 Signed-off-by: luofucong <luofc@foxmail.com> * chore: nix update --------- Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: Ning Sun <sunning@greptime.com>	2025-10-11 07:30:52 +00:00
zyy17	0717773f62	refactor!: add `enable_read_cache` config to support disable read cache explicitly (#6834 ) * refactor: add `enable_read_cache` config to support disable read cache explicitly Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: if `cache_path` is empty and `enable_read_cache` is true, set the default cache dir Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: remove the unessary Option type for `ObjectStorageCacheConfig` Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: sanitize cache config in `DatanodeOptions` and `StandaloneOptions` Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: code review comment Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: apply code review comments Signed-off-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: zyy17 <zyylsxm@gmail.com>	2025-09-26 09:44:12 +00:00
Weny Xu	0c038f755f	refactor(cli): refactor object storage config (#7009 ) * refactor: refactor object storage config Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: public common config Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-09-24 06:50:47 +00:00
Lei, HUANG	9096c5ebbf	chore: bump sequence on region edit (#6947 ) * chore/update-sequence-on-region-edit: ### Commit Message Refactor `get_last_seq_num` Method Across Engines - Change Return Type: Updated the `get_last_seq_num` method to return `Result<SequenceNumber, BoxedError>` instead of `Result<Option<SequenceNumber>, BoxedError>` in the following files: - `src/datanode/src/tests.rs` - `src/file-engine/src/engine.rs` - `src/metric-engine/src/engine.rs` - `src/metric-engine/src/engine/read.rs` - `src/mito2/src/engine.rs` - `src/query/src/optimizer/test_util.rs` - `src/store-api/src/region_engine.rs` - Enhance Region Edit Handling: Modified `RegionWorkerLoop` in `src/mito2/src/worker/handle_manifest.rs` to update file sequences during region edits. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * add committed_sequence to RegionEdit Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: ### Commit Message Refactor sequence retrieval method - Renamed Method: Changed `get_last_seq_num` to `get_committed_sequence` across multiple files to better reflect its purpose of retrieving the latest committed sequence. - Affected files: `tests.rs`, `engine.rs` in `file-engine`, `metric-engine`, `mito2`, `test_util.rs`, and `region_engine.rs`. - Removed Unused Struct: Deleted `RegionSequencesRequest` struct from `region_request.rs` as it is no longer needed. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Add Committed Sequence Handling in Region Engine - `engine.rs`: Introduced a new test module `bump_committed_sequence_test` to verify committed sequence handling. - `bump_committed_sequence_test.rs`: Added a test to ensure the committed sequence is correctly updated and persisted across region reopenings. - `action.rs`: Updated `RegionManifest` and `RegionManifestBuilder` to include `committed_sequence` for tracking. - `manager.rs`: Adjusted manifest size assertion to accommodate new committed sequence data. - `opener.rs`: Implemented logic to override committed sequence during region opening. - `version.rs`: Added `set_committed_sequence` method to update the committed sequence in `VersionControl`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Enhance `test_bump_committed_sequence` in `bump_committed_sequence_test.rs` - Updated the test to include row operations using `build_rows`, `put_rows`, and `rows_schema` to verify the committed sequence behavior. - Adjusted assertions to reflect changes in committed sequence after row operations and region edits. - Added comments to clarify the expected behavior of committed sequence after reopening the region and replaying the WAL. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Enhance Region Sequence Management - `bump_committed_sequence_test.rs`: Updated test to handle region reopening and sequence management, ensuring committed sequences are correctly set and verified after edits. - `opener.rs`: Improved committed sequence handling by overriding it only if the manifest's sequence is greater than the replayed sequence. Added logging for mutation sequence replay. - `region_write_ctx.rs`: Modified `push_mutation` and `push_bulk` methods to adopt sequence numbers from parameters, enhancing sequence management during write operations. - `handle_write.rs`: Updated `RegionWorkerLoop` to pass sequence numbers in `push_bulk` and `push_mutation` methods, ensuring consistent sequence handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: ### Remove Debug Logging from `opener.rs` - Removed debug logging for mutation sequences in `opener.rs` to clean up the output and improve performance. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-09-16 16:22:25 +00:00
Zhenchi	db42ad42dc	feat: add visible to sst entry for staging mode (#6964 ) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-15 09:05:54 +00:00
Zhenchi	264d05d20e	feat: add InformationExtension.inspect_datanode for datanode inspection (#6921 ) * feat: add InformationExtension.inspect_datanode for datanode inspection Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * aggregate results from all datanodes Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix unreleased mito engine Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-09 03:29:04 +00:00
Zhenchi	9fe84f6fbd	feat(mito): backfill partition expr on region open (#6862 ) * feat(mito): backfill partition expr on region open Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * tiny polish Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * only writable leader persists Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * catchup needs backfill Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * handle distribute mode Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * only when transfer to set gracefully Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-09 02:50:01 +00:00
Ning Sun	948a6578fa	feat: add udtf (table function) registration (#6922 ) * feat: work-in-progress udtf support Signed-off-by: Ning Sun <sunning@greptime.com> * feat: add table function support Signed-off-by: Ning Sun <sunning@greptime.com> * test: resolve test error Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com>	2025-09-09 01:26:38 +00:00
Weny Xu	16febbd4c2	feat: add CPU, memory and node status info to `cluster_info` (#6897 ) * feat: add CPU and memory info to `cluster_info` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add `node_status` to `cluster_info` table Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-09-08 08:59:34 +00:00
Ruihang Xia	c9377e7c5a	build: bump rust edition to 2024 (#6920 ) * bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-09-08 02:37:18 +00:00
discord9	f7c8d86ebb	feat: file ref mgr (#6844 ) * feat: file ref manager Signed-off-by: discord9 <discord9@163.com> * refactor: put file ref mgr to sep file Signed-off-by: discord9 <discord9@163.com> * chore: rename path Signed-off-by: discord9 <discord9@163.com> * test: ref path Signed-off-by: discord9 <discord9@163.com> * fix: pass node id Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * feat: exlucde already in manifest files Signed-off-by: discord9 <discord9@163.com> * docs: explain why it can work Signed-off-by: discord9 <discord9@163.com> * feat: also include manifest versions Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * rename func to imply what's it actually doing Signed-off-by: discord9 <discord9@163.com> * more docs Signed-off-by: discord9 <discord9@163.com> * docs: expect gc worker to do the job right Signed-off-by: discord9 <discord9@163.com> * refactor: partially per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: unused Signed-off-by: discord9 <discord9@163.com> * metrics: change to per datanode instead Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-09-05 03:45:01 +00:00
Alex Araujo	2c019965be	feat: unify FlushRegions instructions (#6819 ) * feat: add FlushRegionsV2 instruction with unified semantics - Add FlushRegionsV2 struct supporting both single and batch operations - Preserve original FlushRegion(RegionId) API for backward compatibility - Support configurable FlushStrategy (Sync/Async) and FlushErrorStrategy (FailFast/TryAll) - Add detailed per-region error reporting in FlushRegionReply - Update datanode handlers to support both legacy and enhanced flush instructions - Maintain zero breaking changes through automatic conversion of legacy formats Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * chore: run make fmt Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * Apply suggestions from code review Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com> Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * refactor: extract shared perform_region_flush fn Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * refactor: use consistent error type across similar methods see gh copilot suggestion: https://github.com/GreptimeTeam/greptimedb/pull/6819#discussion_r2299603698 Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * chore: make fmt Signed-off-by: Alex Araujo <alexaraujo@gmail.com> * refactor: consolidate FlushRegion instructions Signed-off-by: Alex Araujo <alexaraujo@gmail.com> --------- Signed-off-by: Alex Araujo <alexaraujo@gmail.com>	2025-09-04 01:18:54 +00:00
Weny Xu	7cf47ccf54	fix: initialize remote WAL regions with correct flushed entry IDs (#6856 ) * fix: initialize remote WAL regions with correct flushed entry IDs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: correct latest offset Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add replay checkpoint to catchup request Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: logs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-09-03 04:00:02 +00:00
Weny Xu	566a647ec7	feat: add replay checkpoint to reduce overhead for remote WAL (#6816 ) * feat: introduce `TopicRegionValue` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: persist region replay checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: udpate config.md Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: send open region instructions with reply checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: use usize Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add topic name pattern Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: enable wal prune by default Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-27 07:24:33 +00:00
Ruihang Xia	474a689309	feat: region prune part 2 (#6752 ) * skeleton Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * get rule set Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adjust style Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * adjust params Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * reuse collider Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * canonize Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * more robust predicate extractor Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify predicate extractor's test and impl Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * unify import Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplification, remove unnecessary interfaces Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle partial referenced exprs Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * finalize predicate extractor Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * document region pruner Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * chore: reduce diff Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify checker Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refine overlapping check method Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * reduce diff Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * coerce types Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove unused errors Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * apply review comment Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor use Bound Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify hashmap Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Apply suggestions from code review Co-authored-by: Yingwen <realevenyag@gmail.com> * sqlness tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * redact region id Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test: update sqlness result after udpate datafusion Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: discord9 <discord9@163.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com>	2025-08-20 18:47:38 +00:00
Weny Xu	021ad09c21	refactor: simplify WAL pruning procedure and introduce region flush trigger (#6741 ) * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: update wal config for metasrv Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce region flush trigger Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: debug assert Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: log level Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify wal prune procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: upgrade rskafka Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: always flush inactive regions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refactor flush trigger Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused code Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: typo Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-14 14:15:30 +00:00
Weny Xu	8659412cac	feat: introduce `PeriodicTopicStatsReporter` (#6730 ) * refactor: introduce `PeriodicTopicStatsReporter` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix typo Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remote wal tests styling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit test Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: handling region wal options not found Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: minor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: upgrade greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-13 11:46:50 +00:00
Zhenchi	dea87b7e57	refactor: use DummyCatalog to construct query engine for datanode (#6723 ) * refactor: use DummyCatalog to construct query engine for datanode Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix clippy Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * move to query/dummy_catalog Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-08-13 09:49:51 +00:00
Ruihang Xia	25f926ea7d	feat: mito region staging state (#6664 ) * fix: not mark all deleted when partial trunc (#6654) * fix: not mark all deleted when partial trunc&not update manifest when partial file range is empty Signed-off-by: discord9 <discord9@163.com> * docs: note Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * some tests and DdlRequest Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * stage transit Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * address CR comments Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * correct error type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>	2025-08-12 03:17:47 +00:00
LFC	253d89b5cc	feat: able to set read preference to flownode (#6696 ) fix: correctly compare the opened follower regions in startup Signed-off-by: luofucong <luofc@foxmail.com>	2025-08-08 09:08:09 +00:00
Weny Xu	c7c8495a6b	feat: add metrics for reconciliation procedures (#6652 ) * feat: add metrics for reconciliation procedures Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: improve error handling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(datanode): handle ignore_nonexistent_region flag in open_all_regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: merge metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-06 11:24:03 +00:00
discord9	1afa0afc67	feat: add partial truncate (#6602 ) * feat: add partial truncate Signed-off-by: discord9 <discord9@163.com> * fix: per review Signed-off-by: discord9 <discord9@163.com> * feat: add proto partial truncate kind Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: update branched proto Signed-off-by: discord9 <discord9@163.com> * feat: grpc support truncate WIP sql support Signed-off-by: discord9 <discord9@163.com> * wip: parse truncate range Signed-off-by: discord9 <discord9@163.com> * feat: truncate by range Signed-off-by: discord9 <discord9@163.com> * fix: truncate range display Signed-off-by: discord9 <discord9@163.com> * chore: resolve todo Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: more invalid parse Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: unused Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: update branch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-08-04 10:50:27 +00:00

1 2 3 4 5 ...

708 Commits