greptimedb

mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-19 06:20:38 +00:00

Author	SHA1	Message	Date
Weny Xu	2f242927a8	feat(repartition): implement region deallocation for repartition procedure (#7522 ) * feat: implement deallocate regions for repartition procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): add force flag to drop physical regions with associated logical regions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: update table metadata after deallocating regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-01-07 06:13:48 +00:00
Weny Xu	b1d81913f5	feat: update `ApplyStagingManifestRequest` to fetch manifest from central region (#7493 ) * feat: update ApplyStagingManifestRequest to fetch manifest from central region Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine comments Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): rename `StagingDataStorage` to `StagingBlobStorage` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-30 07:29:56 +00:00
dennis zhuang	e4b5ef275f	feat: impl vector index building (#7468 ) * feat: impl vector index building Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: supports flat format Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * ci: add vector_index feature to test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions from copilot Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2025-12-30 03:38:51 +00:00
Weny Xu	294f19fa1d	feat(metric-engine): support sync logical regions from source region (#7438 ) * chore: move file Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): support sync logical regions from source region Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-25 09:06:58 +00:00
AntiTopQuark	cea578244c	fix(compaction): unify behavior of database compaction options with TTL (#7402 ) * fix: fix dynamic compactiom option,unify behavior of database compaction options with TTL option Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> * fix unit test Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> * add debug log Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> --------- Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com>	2025-12-25 02:34:42 +00:00
Weny Xu	e1b18614ee	feat(mito2): implement `ApplyStagingManifest` request handling (#7456 ) * feat(mito2): implement `ApplyStagingManifest` request handling Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: fmt Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix logic Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-24 09:05:09 +00:00
discord9	dec0d522f8	feat: gc versioned index (#7412 ) * feat: add index version to file ref Signed-off-by: discord9 <discord9@163.com> * refactor wip Signed-off-by: discord9 <discord9@163.com> * wip Signed-off-by: discord9 <discord9@163.com> * update gc worker Signed-off-by: discord9 <discord9@163.com> * stuff Signed-off-by: discord9 <discord9@163.com> * gc report for index files Signed-off-by: discord9 <discord9@163.com> * fix: type Signed-off-by: discord9 <discord9@163.com> * stuff Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: metrics Signed-off-by: discord9 <discord9@163.com> * typo Signed-off-by: discord9 <discord9@163.com> * typo Signed-off-by: discord9 <discord9@163.com> * chore: naming Signed-off-by: discord9 <discord9@163.com> * docs: update explain Signed-off-by: discord9 <discord9@163.com> * test: parse file id/type from file path Signed-off-by: discord9 <discord9@163.com> * chore: change parse method visibility to crate Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-24 03:07:53 +00:00
dennis zhuang	a35a39f726	feat(vector_index): adds the foundational types and SQL parsing support for vector index (#7366 ) * feat: adds the foundational types and SQL parsing support for vector index Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: by suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: ensure index option values must be greater than zero Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: validate connectivity strictly Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: compile error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: disable SIMD for ci Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2025-12-16 22:45:36 +00:00
Weny Xu	f7d5c87ac0	feat: introduce `copy_region_from` for mito engine (#7389 ) * feat: introduce `copy_region_from` Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-16 06:12:06 +00:00
discord9	f06a64ff90	feat: mark index outdated (#7383 ) * feat: mark index outdated Signed-off-by: discord9 <discord9@163.com> * refactor: move IndexVerwsion to store-api Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * fix: condition for add files Signed-off-by: discord9 <discord9@163.com> * cleanup Signed-off-by: discord9 <discord9@163.com> * refactor(sst): extract index version check into method Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-11 12:08:45 +00:00
discord9	9197e818ec	refactor: use versioned index for index file (#7309 ) * refactor: use versioned index for index file Signed-off-by: discord9 <discord9@163.com> * fix: sst entry table Signed-off-by: discord9 <discord9@163.com> * update sqlness Signed-off-by: discord9 <discord9@163.com> * chore: unit type Signed-off-by: discord9 <discord9@163.com> * fix: missing version Signed-off-by: discord9 <discord9@163.com> * more fix build index Signed-off-by: discord9 <discord9@163.com> * fix: use proper index id Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * test: update Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * test: test_list_ssts fixed Signed-off-by: discord9 <discord9@163.com> * test: fix test Signed-off-by: discord9 <discord9@163.com> * feat: stuff Signed-off-by: discord9 <discord9@163.com> * fix: clean temp index file on abort&delete all index version when delete file Signed-off-by: discord9 <discord9@163.com> * docs: explain Signed-off-by: discord9 <discord9@163.com> * fix: actually clean up tmp dir Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * clean tmp dir only when write cache enabled Signed-off-by: discord9 <discord9@163.com> * refactor: add version to index cache Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * test: update size Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-09 07:31:12 +00:00
Weny Xu	8346acb900	feat: introduce `EnterStagingRequest` for `RegionEngine` (#7261 ) * feat: introduce `EnterStagingRequest` for region engine Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: improve error handling in staging mode entry Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-11-28 09:02:32 +00:00
Sicong Hu	2783a5218e	feat: implement manual type for async index build (#7104 ) * feat: prepare for index_build command Signed-off-by: SNC123 <sinhco@outlook.com> * feat: impl manual index build Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy and fmt Signed-off-by: SNC123 <sinhco@outlook.com> * test: add idempotency check for manual build Signed-off-by: SNC123 <sinhco@outlook.com> * chore: apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * chore: update proto Signed-off-by: SNC123 <sinhco@outlook.com> * chore: apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * chore: fmt Signed-off-by: SNC123 <sinhco@outlook.com> * chore: update proto souce to greptimedb Signed-off-by: SNC123 <sinhco@outlook.com> * fix: cargo.lock Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com>	2025-11-25 15:21:30 +00:00
Weny Xu	6b6d1ce7c4	feat: introduce `remap_manifests` for `RegionEngine` (#7265 ) * refactor: consolidate RegionManifestOptions creation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce`remap_manifests` for `RegionEngine` Signed-off-by: WenyXu <wenymedia@gmail.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-25 12:09:20 +00:00
yihong	d811c4f060	fix: pre-commit all files failed (#7290 ) Signed-off-by: yihong0618 <zouzou0208@gmail.com>	2025-11-25 07:27:46 +00:00
Ruihang Xia	b32ca3ad86	perf: parallelize file source region (#7285 ) Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-11-24 11:37:48 +00:00
Ruihang Xia	25fab2ba7d	feat: don't validate external table's region schema (#7268 ) * feat: don't validate external table's region schema Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-11-21 03:28:14 +00:00
discord9	e59612043d	feat: gc scheduler ctx&procedure (#7252 ) * feat: gc ctx&procedure Signed-off-by: discord9 <discord9@163.com> * fix: handle region not found case Signed-off-by: discord9 <discord9@163.com> * docs: more explain&todo Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: add time for region gc Signed-off-by: discord9 <discord9@163.com> * fix: explain why loader for gc region should fail Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-19 08:35:17 +00:00
discord9	29bbff3c90	feat: gc worker only local regions&test (#7203 ) * feat: gc worker only on local region Signed-off-by: discord9 <discord9@163.com> * more check Signed-off-by: discord9 <discord9@163.com> * chore: stuff Signed-off-by: discord9 <discord9@163.com> * fix: ignore async index file for now Signed-off-by: discord9 <discord9@163.com> * fix: file removal rate calc Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-18 02:45:09 +00:00
Yingwen	bb6a3a2ff3	feat: support altering sst format for a table (#7206 ) * refactor: remove memtable_builder from MitoRegion Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add alter format Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support changing the format and memtable Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support changing sst format via table options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: set scanner and memtable builder with correct format Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix incorrect metadata in version after alter Signed-off-by: evenyag <realevenyag@gmail.com> * test: add sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> * test: replace region_id in sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * fix: create correct memtable when setting sst_format explicitly Signed-off-by: evenyag <realevenyag@gmail.com> * test: sqlness alter_format test set sst_format to primary_key Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove verbose log Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-11-11 13:19:00 +00:00
jeremyhi	c7fded29ee	feat: query mem limiter (#7078 ) * feat: query mem limiter * feat: config docs * feat: frontend query limit config * fix: unused imports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add metrics for query memory tracker Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: right postion for tracker Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid race condition Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: soft and hard limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: when soft_limit == 0 Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: upgrade limit algorithm Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: remove batch window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: batch mem size Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: refine limit algorithm Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: get sys mem Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change * feat: up tracker to the top stream * feat: estimated_size for batch Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor refactor * feat: scan_memory_limit connect to max_concurrent_queries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make callback clearly * feat: add unlimted enum Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment * chore: comment on recursion_limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: refactor and put permit into RegionScanExec Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: multiple lazy static blocks * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2025-11-11 07:47:55 +00:00
Weny Xu	47937961f6	feat(metric)!: enable sparse primary key encoding by default (#7195 ) * feat(metric): enable sparse primary key encoding by default Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update config.md Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * Update src/mito-codec/src/key_values.rs Co-authored-by: Yingwen <realevenyag@gmail.com> * feat: only allow setting primary key encoding for metric engine Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support deleting rows from logical region instead of physical region This keeps the behavior the same as put. It's easier to support sparse encoding for deleting logical regions. Now the metric engine doesn't support delete rows from physical region directly. Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove unused error Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com>	2025-11-11 06:33:51 +00:00
Weny Xu	6960a0183a	refactor: add support for batch region upgrade operations part1 (#7155 ) * refactor: convert UpgradeRegion instruction to batch operation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce `handle_batch_catchup_requests` fn for mito engine Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce `handle_batch_catchup_requests` fn for metric engine Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: suggestion and add ser/de tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-10-31 03:08:38 +00:00
Sicong Hu	30894d7599	feat(mito): Optimize async index building with priority-based batching (#7034 ) * feat: add priority-based batching to IndexBuildScheduler Signed-off-by: SNC123 <sinhco@outlook.com> * fix: clean old puffin-related cache Signed-off-by: SNC123 <sinhco@outlook.com> * test: add test for IndexBuildScheduler Signed-off-by: SNC123 <sinhco@outlook.com> * feat: different index file id for read and async write Signed-off-by: SNC123 <sinhco@outlook.com> * feat: different index file id for delete Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy Signed-off-by: SNC123 <sinhco@outlook.com> * fix: apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * fix: apply comments Signed-off-by: SNC123 <sinhco@outlook.com> * combine files and index files Signed-off-by: SNC123 <sinhco@outlook.com> * feat: add index_file_id into ManifestSstEntry Signed-off-by: SNC123 <sinhco@outlook.com> * Update src/mito2/src/gc.rs Signed-off-by: SNC123 <sinhco@outlook.com> * resolve conflicts Signed-off-by: SNC123 <sinhco@outlook.com> * fix: sqlness Signed-off-by: SNC123 <sinhco@outlook.com> * chore: fmt Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com>	2025-10-31 02:13:17 +00:00
discord9	37bc2e6b07	feat: gc worker heartbeat instruction (#7118 ) again false by default test: config api refactor: per code review less info! even less info!! docs: gc regions instr refactor: grp by region id per code review per review error handling? test: fix todos aft rebase fix after refactor Signed-off-by: discord9 <discord9@163.com>	2025-10-29 02:59:36 +00:00
discord9	68247fc9b1	fix: count_state use stat to eval&predicate w/out region (#7116 ) * fix: count_state use stat to eval Signed-off-by: discord9 <discord9@163.com> * cleanup Signed-off-by: discord9 <discord9@163.com> * fix: use predicate without region Signed-off-by: discord9 <discord9@163.com> * test: diverge standalone/dist impl Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-10-27 02:14:45 +00:00
Yingwen	4c70b4c31d	feat: store estimated series num in file meta (#7126 ) * feat: add num_series to FileMeta Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add SeriesEstimator to collect num_series Signed-off-by: evenyag <realevenyag@gmail.com> * fix: set num_series in compactor Signed-off-by: evenyag <realevenyag@gmail.com> * chore: print num_series in Debug for FileMeta Signed-off-by: evenyag <realevenyag@gmail.com> * style: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: increase series count when next ts <= last Signed-off-by: evenyag <realevenyag@gmail.com> * test: add tests for SeriesEstimator Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add num_series to ssts_manifest table Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness tests Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix metric engine list entry test Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-10-24 05:53:48 +00:00
Sicong Hu	a1af4dce0c	feat: implement three build types for async index build (#7029 ) * feat: impl four types index build Signed-off-by: SNC123 <sinhco@outlook.com> * test: add tests for four types index build Signed-off-by: SNC123 <sinhco@outlook.com> * test: add sqlness test for manual index build Signed-off-by: SNC123 <sinhco@outlook.com> * fix: add region request support and correct sqlness Signed-off-by: SNC123 <sinhco@outlook.com> * fix: update cargo.toml for proto and resolve conflicts Signed-off-by: SNC123 <sinhco@outlook.com> * fix: rebase Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy Signed-off-by: SNC123 <sinhco@outlook.com> * fix: toml fmt and correct sqlness Signed-off-by: SNC123 <sinhco@outlook.com> * fix: correct sqlness result Signed-off-by: SNC123 <sinhco@outlook.com> * refactor: extract manual build logic Signed-off-by: SNC123 <sinhco@outlook.com> * apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * feat: abort index build process Signed-off-by: SNC123 <sinhco@outlook.com> * clippy Signed-off-by: SNC123 <sinhco@outlook.com> * chore: wrap `should_abort_index` Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com>	2025-10-21 02:48:28 +00:00
Zhenchi	938d757523	feat: expose SST index metadata via information schema (#7044 ) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-10-20 11:59:16 +00:00
Zhenchi	7b396bb290	feat(mito2): expose puffin index metadata (#7042 ) * Add encode/decode helpers for IndexTarget Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Use IndexTarget encode for puffin index blob keys Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Normalize puffin index blobs to use IndexTarget keys Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * feat(mito2): expose puffin index metadata Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * target json polish Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix header Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add index path Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address copilot comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * reuse cached index metadata Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * parallelism for reading index meta Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-10-17 06:22:07 +00:00
discord9	ac65ede033	feat: memtable seq range read (#6950 ) * feat: seq range memtable read Signed-off-by: discord9 <discord9@163.com> * test: from&range Signed-off-by: discord9 <discord9@163.com> * wt Signed-off-by: discord9 <discord9@163.com> * after rebase fix Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * docs: better naming&emphaise Signed-off-by: discord9 <discord9@163.com> * refactor: use filter method Signed-off-by: discord9 <discord9@163.com> * tests: unwrap Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-10-16 05:30:56 +00:00
Lei, HUANG	552c502620	feat: manual compaction parallelism (#7086 ) * feat/manual-compaction-parallelism: ### Add Parallelism Support to Compaction Requests - `Cargo.lock` & `Cargo.toml`: Updated `greptime-proto` dependency to a new revision. - `flush_compact_table.rs`: Enhanced `parse_compact_params` to support a new `parallelism` parameter, allowing users to specify the level of parallelism for table compaction. - `handle_compaction.rs`: Integrated `parallelism` into the compaction scheduling process, defaulting to 1 if not specified. - `request.rs` & `region_request.rs`: Modified `CompactRequest` to include `parallelism`, with logic to handle unspecifie values. - `requests.rs`: Updated `CompactTableRequest` structure to include an optional `parallelism` field. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/manual-compaction-parallelism: ### Commit Message Enhance Compaction Request Handling - `flush_compact_table.rs`: - Renamed `parse_compact_params` to `parse_compact_request`. - Introduced `DEFAULT_COMPACTION_PARALLELISM` constant. - Updated parsing logic to handle keyword arguments for `strict_window` and `regular` compaction types, including `parallelism` and `window`. - Modified tests to reflect changes in parsing logic and default parallelism handling. - `request.rs`: - Updated `parallelism` handling in `RegionRequestBody::Compact` to use the new default value. - `requests.rs`: - Changed `CompactTableRequest` to use a non-optional `parallelism` field with a default value of 1. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/manual-compaction-parallelism: ### Update `flush_compact_table.rs` Parameter Validation - Modified parameter validation in `flush_compact_table.rs` to restrict the maximum number of parameters from 4 to 3 in the `parse_compact_request` function. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/manual-compaction-parallelism: Update `greptime-proto` dependency - Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-10-16 03:47:01 +00:00
Yingwen	a9c342b0f7	feat: support setting sst_format in table options (#7068 ) * feat: add FormatType to support multi format in the future Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add sst_format to RegionOptions Signed-off-by: evenyag <realevenyag@gmail.com> * feat: sets the sst_format based on RegionOptions Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add sst_format to mito table options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix RegionManifest deserialization without sst_format Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove Parquet suffix from FormatType Signed-off-by: evenyag <realevenyag@gmail.com> * feat: prefer RegionOptions::sst_format in compactor/memtable builder Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename enable_experimental_flat_format to default_experimental_flat_format Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config.md Signed-off-by: evenyag <realevenyag@gmail.com> * style: fmt Signed-off-by: evenyag <realevenyag@gmail.com> * test: update manifest test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors, handle sst_format in remap_manifest Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-10-13 08:38:37 +00:00
LFC	8fe17d43d5	chore: update rust to nightly 2025-10-01 (#7069 ) * chore: update rust to nightly 2025-10-01 Signed-off-by: luofucong <luofc@foxmail.com> * chore: nix update --------- Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: Ning Sun <sunning@greptime.com>	2025-10-11 07:30:52 +00:00
discord9	18e4839a17	feat: datanode side local gc worker (#6940 ) * wip Signed-off-by: discord9 <discord9@163.com> * docs for behavior Signed-off-by: discord9 <discord9@163.com> * wip: handle outdated version Signed-off-by: discord9 <discord9@163.com> * feat: just retry Signed-off-by: discord9 <discord9@163.com> * feat: smaller lingering time Signed-off-by: discord9 <discord9@163.com> * refactor: partial per review Signed-off-by: discord9 <discord9@163.com> * refactor: rm tmp file cnt Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: opt partial Signed-off-by: discord9 <discord9@163.com> * chore: rebase fix Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-09-19 03:20:13 +00:00
discord9	e26b98f452	refactor: put FileId to store-api (#6988 ) * refactor: put FileId to store-api Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: lock file Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-09-18 03:20:42 +00:00
Lei, HUANG	9096c5ebbf	chore: bump sequence on region edit (#6947 ) * chore/update-sequence-on-region-edit: ### Commit Message Refactor `get_last_seq_num` Method Across Engines - Change Return Type: Updated the `get_last_seq_num` method to return `Result<SequenceNumber, BoxedError>` instead of `Result<Option<SequenceNumber>, BoxedError>` in the following files: - `src/datanode/src/tests.rs` - `src/file-engine/src/engine.rs` - `src/metric-engine/src/engine.rs` - `src/metric-engine/src/engine/read.rs` - `src/mito2/src/engine.rs` - `src/query/src/optimizer/test_util.rs` - `src/store-api/src/region_engine.rs` - Enhance Region Edit Handling: Modified `RegionWorkerLoop` in `src/mito2/src/worker/handle_manifest.rs` to update file sequences during region edits. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * add committed_sequence to RegionEdit Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: ### Commit Message Refactor sequence retrieval method - Renamed Method: Changed `get_last_seq_num` to `get_committed_sequence` across multiple files to better reflect its purpose of retrieving the latest committed sequence. - Affected files: `tests.rs`, `engine.rs` in `file-engine`, `metric-engine`, `mito2`, `test_util.rs`, and `region_engine.rs`. - Removed Unused Struct: Deleted `RegionSequencesRequest` struct from `region_request.rs` as it is no longer needed. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Add Committed Sequence Handling in Region Engine - `engine.rs`: Introduced a new test module `bump_committed_sequence_test` to verify committed sequence handling. - `bump_committed_sequence_test.rs`: Added a test to ensure the committed sequence is correctly updated and persisted across region reopenings. - `action.rs`: Updated `RegionManifest` and `RegionManifestBuilder` to include `committed_sequence` for tracking. - `manager.rs`: Adjusted manifest size assertion to accommodate new committed sequence data. - `opener.rs`: Implemented logic to override committed sequence during region opening. - `version.rs`: Added `set_committed_sequence` method to update the committed sequence in `VersionControl`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Enhance `test_bump_committed_sequence` in `bump_committed_sequence_test.rs` - Updated the test to include row operations using `build_rows`, `put_rows`, and `rows_schema` to verify the committed sequence behavior. - Adjusted assertions to reflect changes in committed sequence after row operations and region edits. - Added comments to clarify the expected behavior of committed sequence after reopening the region and replaying the WAL. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: Enhance Region Sequence Management - `bump_committed_sequence_test.rs`: Updated test to handle region reopening and sequence management, ensuring committed sequences are correctly set and verified after edits. - `opener.rs`: Improved committed sequence handling by overriding it only if the manifest's sequence is greater than the replayed sequence. Added logging for mutation sequence replay. - `region_write_ctx.rs`: Modified `push_mutation` and `push_bulk` methods to adopt sequence numbers from parameters, enhancing sequence management during write operations. - `handle_write.rs`: Updated `RegionWorkerLoop` to pass sequence numbers in `push_bulk` and `push_mutation` methods, ensuring consistent sequence handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/update-sequence-on-region-edit: ### Remove Debug Logging from `opener.rs` - Removed debug logging for mutation sequences in `opener.rs` to clean up the output and improve performance. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-09-16 16:22:25 +00:00
Zhenchi	db42ad42dc	feat: add visible to sst entry for staging mode (#6964 ) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-15 09:05:54 +00:00
Zhenchi	21ee981b49	feat: add origin_region_id and node_id to sst entry (#6937 ) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-09 09:45:15 +00:00
Ruihang Xia	c9377e7c5a	build: bump rust edition to 2024 (#6920 ) * bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-09-08 02:37:18 +00:00
Weny Xu	658d07bfc8	feat: add `written_bytes_since_open` column to `region_statistics` table (#6904 ) * feat: add `write_bytes` column to `region_statistics` table Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename `write_bytes` to `written_bytes` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename `written_bytes` to `written_bytes_since_open` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-09-05 07:27:30 +00:00
Weny Xu	7cf47ccf54	fix: initialize remote WAL regions with correct flushed entry IDs (#6856 ) * fix: initialize remote WAL regions with correct flushed entry IDs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: correct latest offset Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add replay checkpoint to catchup request Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: logs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-09-03 04:00:02 +00:00
Zhenchi	9175fa643d	feat: add schema and recordbatch builder for sst entry (#6841 ) * feat: add schema and recordbatch builder for sst entry Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add build plan Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-09-01 12:46:49 +00:00
Weny Xu	566a647ec7	feat: add replay checkpoint to reduce overhead for remote WAL (#6816 ) * feat: introduce `TopicRegionValue` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: persist region replay checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: udpate config.md Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: send open region instructions with reply checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: use usize Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add topic name pattern Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: enable wal prune by default Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-27 07:24:33 +00:00
Arshdeep	8894cb5406	feat: resolve unused dependencies with cargo-udeps (#6578 ) (#6619 ) * feat:resolve unused dependencies with cargo-udeps (#6578) Signed-off-by: Arshdeep54 <balarsh535@gmail.com> * Apply suggestion from @zyy17 Co-authored-by: zyy17 <zyylsxm@gmail.com> * Apply suggestion from @zyy17 Co-authored-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: Arshdeep54 <balarsh535@gmail.com> Co-authored-by: Ning Sun <classicning@gmail.com> Co-authored-by: zyy17 <zyylsxm@gmail.com>	2025-08-26 10:22:53 +00:00
dennis zhuang	3014972202	chore: improve error message when there are more than one time index (#6796 ) * chore: improve error message when there are more than one time index Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: style Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2025-08-22 09:22:55 +00:00
Zhenchi	6839b5aef4	feat(mito): list SSTs from manifest and storage (#6766 ) * feat(engine): list SSTs from manifest and storage Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * mito engine only Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * try best to remove allocation Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>	2025-08-22 09:22:42 +00:00
Weny Xu	896d72191e	feat: introduce `PersistStatsHandler` (#6777 ) * feat: add `Inserter` trait and impl Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: import items Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce `PersistStatsHandler` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: disable persisting stats in sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * reset channel manager Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: avoid to collect Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove insert options Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use `write_bytes` instead of `write_bytes_per_sec` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: compute write bytes delta Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * Update src/meta-srv/src/handler/persist_stats_handler.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com>	2025-08-21 09:34:58 +00:00
Zhenchi	fb3b1d4866	feat: Store partition expr in RegionMetadata (#6699 ) * wire partition.expr_json option constant and parsing test(mito2): manifest roundtrip persists partition_expr JSON test(mito2): create/open with partition.expr_json persists in manifest docs: add comments for partition.expr_json option and RegionOptions.partition_expr serde: include RegionOptions.partition_expr (skip if None) test(mito2): doc intent and verify runtime backfill + persistence-after-alter for partition expr add partition expr to create request Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add create_with_partition_expr_persists_manifest Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * pass partition expr to create request Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix sqlness Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * remove unused dep Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com>	2025-08-13 22:05:37 +00:00
Weny Xu	8659412cac	feat: introduce `PeriodicTopicStatsReporter` (#6730 ) * refactor: introduce `PeriodicTopicStatsReporter` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix typo Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remote wal tests styling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit test Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: handling region wal options not found Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: minor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: upgrade greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-08-13 11:46:50 +00:00

1 2 3 4 5 ...

370 Commits