greptimedb

mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-24 08:50:40 +00:00

Author	SHA1	Message	Date
dennis zhuang	a56a00224f	feat: impl vector index scan in storage (#7528 ) * feat: impl vector index scan in storage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: fallback to read remote blob when blob not found Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: refactor encoding and decoding and apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: license Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add apply_with_k tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: forgot to align nulls when the vector column is not in the batch Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: add test for vector column is not in a batch while buiilding Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2026-01-12 08:30:51 +00:00
discord9	6487f14f70	feat: gc schd update repart mapping (#7517 ) * feat(gc): batch gc now alos handle routing Signed-off-by: discord9 <discord9@163.com> typo Signed-off-by: discord9 <discord9@163.com> s Signed-off-by: discord9 <discord9@163.com> feat: use batch gc procedure Signed-off-by: discord9 <discord9@163.com> feat: cross region refs Signed-off-by: discord9 <discord9@163.com> feat: clean up repartition Signed-off-by: discord9 <discord9@163.com> chore: cleanup Signed-off-by: discord9 <discord9@163.com> per review Signed-off-by: discord9 <discord9@163.com> test: update mock test Signed-off-by: discord9 <discord9@163.com> refactor: rm unused Signed-off-by: discord9 <discord9@163.com> refactor: invert related_regions Signed-off-by: discord9 <discord9@163.com> clippy Signed-off-by: discord9 <discord9@163.com> pcr Signed-off-by: discord9 <discord9@163.com> chore: remove unused Signed-off-by: discord9 <discord9@163.com> fix: after invert fix Signed-off-by: discord9 <discord9@163.com> chore: rm unused Signed-off-by: discord9 <discord9@163.com> refactor: eff Signed-off-by: discord9 <discord9@163.com> docs: chore Signed-off-by: discord9 <discord9@163.com> * after rebase fix Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * fix: mssing region Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-01-12 08:28:34 +00:00
Yingwen	ef6dd5b99f	fix: precise filter time index if not in projection (#7531 ) * fix: precise filter time index if not in projection Signed-off-by: evenyag <realevenyag@gmail.com> * test: add sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2026-01-07 11:15:34 +00:00
Ruihang Xia	d39895a970	feat: tune query traces (#7524 ) * feat: add partition and region id Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * wip: instrument mito Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * connect region scan span Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * instrument streams Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * tweak Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2026-01-07 08:11:09 +00:00
Weny Xu	2f242927a8	feat(repartition): implement region deallocation for repartition procedure (#7522 ) * feat: implement deallocate regions for repartition procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): add force flag to drop physical regions with associated logical regions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: update table metadata after deallocating regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2026-01-07 06:13:48 +00:00
discord9	6f86a22e6f	feat: adjust some args to gc worker (#7469 ) * chore: less stuff sent Signed-off-by: discord9 <discord9@163.com> * after rebase fix Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * fix: clarify comment on manifest file removal for GC worker Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2026-01-06 07:37:05 +00:00
Weny Xu	b1d81913f5	feat: update `ApplyStagingManifestRequest` to fetch manifest from central region (#7493 ) * feat: update ApplyStagingManifestRequest to fetch manifest from central region Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine comments Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): rename `StagingDataStorage` to `StagingBlobStorage` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-30 07:29:56 +00:00
dennis zhuang	e4b5ef275f	feat: impl vector index building (#7468 ) * feat: impl vector index building Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: supports flat format Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * ci: add vector_index feature to test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions from copilot Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com>	2025-12-30 03:38:51 +00:00
Lei, HUANG	7bc0934eb3	refactor(mito2): make MemtableStats fields public (#7488 ) Change visibility of estimated_bytes, time_range, max_sequence, and series_count fields from private to public for external access. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-12-26 09:57:18 +00:00
Yingwen	89b9469250	feat: Implement per range stats for bulk memtable (#7486 ) * feat: implement per range stats for MemtableRange Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: extract methods to MemtableRanges Signed-off-by: evenyag <realevenyag@gmail.com> * fix: simple bulk memtable set other fields in stats Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use time_index_type() Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use time index type Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-26 07:24:11 +00:00
Weny Xu	518a4e013b	refactor(mito2): reorganize manifest storage into modular components (#7483 ) * refactor(mito2): reorganize manifest storage into modular components Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: sort Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: fmt Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-26 02:24:27 +00:00
Weny Xu	294f19fa1d	feat(metric-engine): support sync logical regions from source region (#7438 ) * chore: move file Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): support sync logical regions from source region Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-25 09:06:58 +00:00
discord9	aea4e9fa55	fix: RemovedFiles deser compatibility (#7475 ) * fix: compat for RemovedFiles Signed-off-by: discord9 <discord9@163.com> * cr Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-25 02:50:34 +00:00
AntiTopQuark	cea578244c	fix(compaction): unify behavior of database compaction options with TTL (#7402 ) * fix: fix dynamic compactiom option,unify behavior of database compaction options with TTL option Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> * fix unit test Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> * add debug log Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com> --------- Signed-off-by: AntiTopQuark <AntiTopQuark1350@outlook.com>	2025-12-25 02:34:42 +00:00
Weny Xu	e1b18614ee	feat(mito2): implement `ApplyStagingManifest` request handling (#7456 ) * feat(mito2): implement `ApplyStagingManifest` request handling Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: fmt Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix logic Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-24 09:05:09 +00:00
Weny Xu	2d9967b981	fix(mito2): pass partition expr explicitly to flush task for region (#7461 ) * fix(mito2): pass partition expr explicitly to flush task for staging mode Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-24 04:18:06 +00:00
discord9	dec0d522f8	feat: gc versioned index (#7412 ) * feat: add index version to file ref Signed-off-by: discord9 <discord9@163.com> * refactor wip Signed-off-by: discord9 <discord9@163.com> * wip Signed-off-by: discord9 <discord9@163.com> * update gc worker Signed-off-by: discord9 <discord9@163.com> * stuff Signed-off-by: discord9 <discord9@163.com> * gc report for index files Signed-off-by: discord9 <discord9@163.com> * fix: type Signed-off-by: discord9 <discord9@163.com> * stuff Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: metrics Signed-off-by: discord9 <discord9@163.com> * typo Signed-off-by: discord9 <discord9@163.com> * typo Signed-off-by: discord9 <discord9@163.com> * chore: naming Signed-off-by: discord9 <discord9@163.com> * docs: update explain Signed-off-by: discord9 <discord9@163.com> * test: parse file id/type from file path Signed-off-by: discord9 <discord9@163.com> * chore: change parse method visibility to crate Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-24 03:07:53 +00:00
discord9	b3bc3c76f1	feat: file range dynamic filter (#7441 ) * feat: add dynamic filtering support in file range and predicate handling Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-23 06:15:30 +00:00
Lei, HUANG	a8b512dded	chore: expose symbols (#7451 ) * chore/expose-symbols: ### Commit Message Enhance `merge_and_dedup` Functionality in `flush.rs` - Function Signature Update: Modified the `merge_and_dedup` function to accept `append_mode` and `merge_mode` as separate parameters instead of using `options`. - Function Accessibility: Changed the visibility of `merge_and_dedup` to `pub` to allow external access. - Function Calls Update: Updated calls to `merge_and_dedup` within `memtable_flat_sources` to align with the new function signature, passing `options.append_mode` and `options.merge_mode()` directly. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/expose-symbols: ### Add Merge and Deduplication Functionality - File: `src/mito2/src/flush.rs` - Introduced `merge_and_dedup` function to merge multiple record batch iterators and apply deduplication based on specified modes. - Added detailed documentation for the function, explaining its arguments, behavior, and usage examples. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-12-22 05:39:03 +00:00
Yingwen	fed6cb0806	fix: flat format use correct encoding in indexer for tags (#7440 ) * test: add inverted and skipping test Signed-off-by: evenyag <realevenyag@gmail.com> * test: Add tests for fulltext index Signed-off-by: evenyag <realevenyag@gmail.com> * fix: index dictionary type in correct encoding in flat format Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use encode_data_type() in SortField Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: refine imports Signed-off-by: evenyag <realevenyag@gmail.com> * test: add tests for sparse encoding Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove logs Signed-off-by: evenyag <realevenyag@gmail.com> * test: update list test Signed-off-by: evenyag <realevenyag@gmail.com> * test: simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-19 07:36:44 +00:00
Lanqing Yang	658332fe68	chore(mito): nit remove extra hashset in gc workers (#7399 ) chore(mito): remove extra hashset in gc workers Signed-off-by: lyang24 <lanqingy93@gmail.com>	2025-12-18 13:09:32 +00:00
jeremyhi	95eccd6cde	feat: introduce granularity for memory manager (#7416 ) * feat: introduce granularity for memory manager Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: add unit test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: remove granularity getter for mamanger Signed-off-by: jeremyhi <fengjiachun@gmail.com> * Update src/common/memory-manager/src/manager.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> * feat: acquire_with_policy for manager Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>	2025-12-17 11:08:51 +00:00
Lei, HUANG	da964880f5	chore: expose symbols (#7417 ) * refactor/expose-symbols: ## Refactor `bulk/part.rs` to Simplify Mutation Handling - Removed the `mutations_to_record_batch` function and its associated helper functions, including `ArraysSorter`, `timestamp_array_to_iter`, and `binary_array_to_dictionary`, to simplify the mutation handling logic in `bulk/part.rs`. - Deleted related test functions `check_binary_array_to_dictionary` and `check_mutations_to_record_batches` from the test module, along with their associated test cases. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor/expose-symbols: ### Commit Message Refactor and Enhance Deduplication Logic - `flush.rs`: Refactored `maybe_dedup_one` function to accept `append_mode` and `merge_mode` as parameters instead of `RegionOptions`. This change enhances flexibility in deduplication logic. - `memtable/bulk.rs`: Made `BulkRangeIterBuilder` struct and its fields public to allow external access and modification, improving extensibility. - `sst.rs`: Corrected a typo in the schema documentation, changing `__prmary_key` to `__primary_key` for clarity and accuracy. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-12-17 01:29:36 +00:00
Yingwen	f6afb10e33	feat!: download file to fill the cache on write cache miss (#7294 ) * feat: download inverted index file Signed-off-by: evenyag <realevenyag@gmail.com> * feat: download for bloom and fulltext Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement maybe_download_background for FileCache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: load file for parquet Signed-off-by: evenyag <realevenyag@gmail.com> * chore: reduce channel size Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use ManifestCache Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass cache to ManifestObjectStore::new Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix fmt and clippy Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove manifest cache ttl Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove read cache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clean old read cache path Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config examples Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix CI Signed-off-by: evenyag <realevenyag@gmail.com> * fix: also clean the root directory Signed-off-by: evenyag <realevenyag@gmail.com> * test: update manifest test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * feat: skip file if it exists Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove warn in replace Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add a flag to enable/disable background download set the concurrency to 1 for background download Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename write_cache_enable_background_download to enable_refill_cache_on_read Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address comments Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update config.md Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-16 08:31:26 +00:00
Weny Xu	f7d5c87ac0	feat: introduce `copy_region_from` for mito engine (#7389 ) * feat: introduce `copy_region_from` Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-16 06:12:06 +00:00
jeremyhi	32f9cc5286	feat: move memory_manager to common crate (#7408 ) * feat: move memory_manager to common crate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: add license header Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by AI comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>	2025-12-15 13:15:33 +00:00
Yingwen	5232a12a8c	feat: per file scan metrics (#7396 ) * feat: collect per file metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: divide build_cost to build_part_cost and build_reader_cost Signed-off-by: evenyag <realevenyag@gmail.com> * feat: limit the file metrics num to display Signed-off-by: evenyag <realevenyag@gmail.com> * fix: use sorted iter to get sorted files Signed-off-by: evenyag <realevenyag@gmail.com> * fix: output metrics in desc order Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-15 12:52:03 +00:00
jeremyhi	baffed8c6a	feat: mem manager on compaction (#7305 ) * feat: mem manager on compaction Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by copilot review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: experimental_ Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: refine estimate_compaction_bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: make them into config example Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by copilot comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * Update src/mito2/src/compaction.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> * fix: dedup the regions waiting Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add AdditionalMemoryGuard for the running compaction task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: do OnExhaustedPolicy before running task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: use OwnedSemaphorePermit to impl guard Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add early_release_partial method to release a portion of memory Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: 0 bytes make request_additional unlimited Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: fail-fast on acquire Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>	2025-12-12 06:49:58 +00:00
Lanqing Yang	f5e0e94e3a	chore(mito): nit avoid clone the batch object on inverted index building (#7388 ) fix: avoid clone the batch object on inverted index building Signed-off-by: lyang24 <lanqingy93@gmail.com>	2025-12-12 04:58:37 +00:00
discord9	f06a64ff90	feat: mark index outdated (#7383 ) * feat: mark index outdated Signed-off-by: discord9 <discord9@163.com> * refactor: move IndexVerwsion to store-api Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * fix: condition for add files Signed-off-by: discord9 <discord9@163.com> * cleanup Signed-off-by: discord9 <discord9@163.com> * refactor(sst): extract index version check into method Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-11 12:08:45 +00:00
discord9	a26dee0ca1	fix: gc listing op first (#7385 ) Signed-off-by: discord9 <discord9@163.com>	2025-12-11 03:25:05 +00:00
Yingwen	a22d08f1b1	feat: collect merge and dedup metrics (#7375 ) * feat: collect FlatMergeReader metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add MergeMetricsReporter, rename Metrics to MergeMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove num_input_rows from MergeMetrics The merge reader won't dedup so there is no need to collect input rows Signed-off-by: evenyag <realevenyag@gmail.com> * feat: report merge metrics to PartitionMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add dedup cost to DedupMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect dedup metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove metrics from FlatMergeIterator Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove num_output_rows from MergeMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement merge() for merge and dedup metrics Signed-off-by: evenyag <realevenyag@gmail.com> * fix: report metrics after observe metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-10 09:16:20 +00:00
Lei, HUANG	2f9130a2de	chore(mito): expose some symbols (#7373 ) chore/expose-symbols: ### Commit Summary - Visibility Changes: Updated visibility of functions in `bulk/part.rs`: - Made `record_batch_estimated_size` and `sort_primary_key_record_batch` functions public. - Enhancements: Enhanced functionality in `memtable.rs` by exposing additional components from `bulk::part`: - `BulkPartEncoder`, `BulkPartMeta`, `UnorderedPart`, `record_batch_estimated_size`, and `sort_primary_key_record_batch`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>	2025-12-09 14:33:14 +00:00
discord9	9197e818ec	refactor: use versioned index for index file (#7309 ) * refactor: use versioned index for index file Signed-off-by: discord9 <discord9@163.com> * fix: sst entry table Signed-off-by: discord9 <discord9@163.com> * update sqlness Signed-off-by: discord9 <discord9@163.com> * chore: unit type Signed-off-by: discord9 <discord9@163.com> * fix: missing version Signed-off-by: discord9 <discord9@163.com> * more fix build index Signed-off-by: discord9 <discord9@163.com> * fix: use proper index id Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * test: update Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * test: test_list_ssts fixed Signed-off-by: discord9 <discord9@163.com> * test: fix test Signed-off-by: discord9 <discord9@163.com> * feat: stuff Signed-off-by: discord9 <discord9@163.com> * fix: clean temp index file on abort&delete all index version when delete file Signed-off-by: discord9 <discord9@163.com> * docs: explain Signed-off-by: discord9 <discord9@163.com> * fix: actually clean up tmp dir Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * clean tmp dir only when write cache enabled Signed-off-by: discord9 <discord9@163.com> * refactor: add version to index cache Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * test: update size Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-12-09 07:31:12 +00:00
Ruihang Xia	edb1f6086f	feat: decode pk eagerly (#7350 ) * feat: decode pk eagerly Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * merge primary_key_codec and decode_primary_key_values Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-12-05 09:11:51 +00:00
Yingwen	84e4e42ee7	feat: add more verbose metrics to scanners (#7336 ) * feat: add inverted applier metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add metrics to bloom applier Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add metrics to fulltext index applier Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement BloomFilterReadMetrics for BloomFilterReader Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect read metrics for inverted index Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add metrics for range_read and metadata Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename elapsed to fetch_elapsed Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect metadata fetch metrics for inverted index Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect cache metrics for inverted and bloom index Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect read metrics in appliers Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect fulltext dir metrics for applier Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect parquet row group metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add parquet metadata metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add apply metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect more metrics for memory row group Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add fetch metrics to ReaderMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: init verbose metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: debug print metrics in ScanMetricsSet Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement debug for new metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * feat: update parquet fetch metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect the whole fetch time Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add file_scan_cost Signed-off-by: evenyag <realevenyag@gmail.com> * chore: parquet fetch add cache_miss counter Signed-off-by: evenyag <realevenyag@gmail.com> * feat: print index read metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: use actual bytes to increase counter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove provided implementations for index reader traits Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: change get_parquet_meta_data() method to receive metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename file_scan_cost to sst_scan_cost Signed-off-by: evenyag <realevenyag@gmail.com> * chore: refine ParquetFetchMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * style: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove useless inner method Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: collect page size actual needed Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify InvertedIndexReadMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplfy InvertedIndexApplyMetrics Debug Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify BloomFilterReadMetrics Debug Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify BloomFilterIndexApplyMetrics Debug Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify FulltextIndexApplyMetrics implementation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify ParquetFetchMetrics Debug Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify MetadataCacheMetrics Debug Signed-off-by: evenyag <realevenyag@gmail.com> * feat: only print verbose metrics when they are not empty. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use mutex to protect ParquetFetchMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * style: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use duration for elapsed in ParquetFetchMetricsData Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-04 13:40:18 +00:00
Yingwen	d5c616a9ff	feat: implement a cache for manifest files (#7326 ) * feat: use cache in manifest store Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use ManifestCache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clean empty manifest dir Signed-off-by: evenyag <realevenyag@gmail.com> * feat: get last checkpoint from cache Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add hit/miss counter for manifest cache Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add logs Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass cache to ManifestObjectStore::new Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * feat: cache checkpoint Signed-off-by: evenyag <realevenyag@gmail.com> * feat: cache checkpoint in write Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler warnings Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update config comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: manifest store cache for staging Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: move recover_inner to FileCacheInner Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove manifest cache config from MitoConfig Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: reduce clone when cache is enabled Signed-off-by: evenyag <realevenyag@gmail.com> * feat: do not cache staging manifests We clean staging manifests by remove_all which isn't easy to clean the cache in the same way Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix paths in manifest cache Signed-off-by: evenyag <realevenyag@gmail.com> * chore: don't clean dir if it is too new Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: reuse write cache ttl as manifest cache ttl Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: clean all empty subdirectories Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-12-04 12:51:09 +00:00
Weny Xu	0177f244e9	fix: fix write stall that never recovers due to flush logic issues (#7322 ) * fix: fix write stall that never recovers due to flush logic issues Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit test Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: flush multiple regions when engine is full Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine fn name Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify flush scheduler by removing flushing state Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-12-02 12:48:41 +00:00
Weny Xu	8346acb900	feat: introduce `EnterStagingRequest` for `RegionEngine` (#7261 ) * feat: introduce `EnterStagingRequest` for region engine Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: improve error handling in staging mode entry Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>	2025-11-28 09:02:32 +00:00
LFC	fdab75ce27	feat: simple read write new json type values (#7175 ) feat: basic json read and write Signed-off-by: luofucong <luofc@foxmail.com>	2025-11-27 12:40:35 +00:00
Yingwen	afefc0c604	fix: implement bulk write for time partitions and bulk memtable (#7293 ) * feat: implement convert_bulk_part Signed-off-by: evenyag <realevenyag@gmail.com> * chore: convert bulk part in TimePartitions Signed-off-by: evenyag <realevenyag@gmail.com> * feat: fill missing columns for bulk parts Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comments Signed-off-by: evenyag <realevenyag@gmail.com> * feat: cast to dictionary type Signed-off-by: evenyag <realevenyag@gmail.com> * test: add unit tests Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: do not convert part if bulk is written by write() Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>	2025-11-27 08:01:45 +00:00
discord9	0aeaf405c7	feat: add batch gc procedure (#7296 ) * feat: add batch gc procedure Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * per even review Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-27 03:58:15 +00:00
Yingwen	b5cbc35a0d	fix: partition tree metric should the delta (#7307 ) Signed-off-by: evenyag <realevenyag@gmail.com>	2025-11-27 03:49:02 +00:00
discord9	6485a26fa3	refactor: load metadata using offical impl (#7302 ) * refactor: load metadata using offical impl Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-26 08:52:04 +00:00
Sicong Hu	2783a5218e	feat: implement manual type for async index build (#7104 ) * feat: prepare for index_build command Signed-off-by: SNC123 <sinhco@outlook.com> * feat: impl manual index build Signed-off-by: SNC123 <sinhco@outlook.com> * chore: clippy and fmt Signed-off-by: SNC123 <sinhco@outlook.com> * test: add idempotency check for manual build Signed-off-by: SNC123 <sinhco@outlook.com> * chore: apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * chore: update proto Signed-off-by: SNC123 <sinhco@outlook.com> * chore: apply suggestions Signed-off-by: SNC123 <sinhco@outlook.com> * chore: fmt Signed-off-by: SNC123 <sinhco@outlook.com> * chore: update proto souce to greptimedb Signed-off-by: SNC123 <sinhco@outlook.com> * fix: cargo.lock Signed-off-by: SNC123 <sinhco@outlook.com> --------- Signed-off-by: SNC123 <sinhco@outlook.com>	2025-11-25 15:21:30 +00:00
Weny Xu	6b6d1ce7c4	feat: introduce `remap_manifests` for `RegionEngine` (#7265 ) * refactor: consolidate RegionManifestOptions creation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce`remap_manifests` for `RegionEngine` Signed-off-by: WenyXu <wenymedia@gmail.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2025-11-25 12:09:20 +00:00
yihong	d811c4f060	fix: pre-commit all files failed (#7290 ) Signed-off-by: yihong0618 <zouzou0208@gmail.com>	2025-11-25 07:27:46 +00:00
Ruihang Xia	b32ca3ad86	perf: parallelize file source region (#7285 ) Signed-off-by: Ruihang Xia <waynestxia@gmail.com>	2025-11-24 11:37:48 +00:00
discord9	52a576cf6d	feat: basic gc scheduler (#7263 ) * feat: basic gc scheduler Signed-off-by: discord9 <discord9@163.com> * refactor: rm dup code Signed-off-by: discord9 <discord9@163.com> * docs: todo for cleaner code Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * feat: rm retry path Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * feat: skip first full listing after metasrv start Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>	2025-11-24 07:57:18 +00:00
LFC	4a7c16586b	refactor: remove `Vector`s from `RecordBatch` completely (#7184 ) * refactor: remove `Vector`s from `RecordBatch` completely Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com>	2025-11-21 08:53:35 +00:00

1 2 3 4 5 ...

794 Commits