* feat: clear the range builder after one part
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: collect peak memory usage of build ranges
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: collect peak range builder nums in metrics
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: num_range_builders_peak -> num_peak_range_builders
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: track file range counts
* Ensure the reader won't be released until all ranges scanned.
* This fixes unordered scan which each partition range is a row group
Signed-off-by: evenyag <realevenyag@gmail.com>
* style: fix clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: change to isize
The metrics may init to 0.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: flush region before close when skip-wal is enabled
When closing a region with Noop WAL provider, the region is now flushed
before closing to ensure data durability. This prevents data loss for
regions configured with skip_wal.
Changes:
- Add `Closing` variant to `FlushReason` enum
- Modify `handle_close_request` to trigger flush for Noop WAL regions
- Pass flush reason through the flush pipeline
- Add test to verify data persistence after close with skip-wal
The flush-on-close flow completes the region cleanup after the flush
finishes, ensuring the region is properly removed from all schedulers.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: extract region cleanup logic into dedicated method
Extracts common region cleanup logic (stop, remove, and scheduler cleanup) into a new `remove_region` method to avoid duplication between `handle_close` and `handle_flush_request`. This improves code maintainability and reduces redundancy.
Also updates `RegionMap::remove_region` to return the removed region reference, allowing the caller to perform cleanup operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test: split skip-wal region close test into pending and no-pending cases
Split the test_close_region_skip_wal test into two separate test cases:
- test_close_region_skip_wal_with_pending_data: Tests the scenario where
data is inserted before closing a region with skip-wal enabled
- test_close_region_skip_wal_without_pending_data: Tests the scenario
where a region with skip-wal is closed without any data insertion
This improves test clarity and ensures both scenarios are properly covered.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip request handling and compaction for flush-on-close regions
When a region is flushed as part of the close operation (flush_on_close=true),
the region is immediately removed from the server. Therefore, there's no need
to handle pending requests or schedule compactions for such regions.
This fix moves the on_flush_success listener call outside the conditional
block and wraps all post-flush operations (request handling, compaction
scheduling) in an else branch, ensuring they only execute for normal flush
operations where the region remains active.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test: add close follower region test with skip-wal
Adds a test case for closing a follower region with skip-wal enabled.
The test verifies that when a region transitions from Follower to Leader
before closing, the flush mechanism works correctly even with WAL disabled.
Also refactors flushable_region() to return Option instead of erroring
when region is not operable, allowing more flexible handling of region
states during flush operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: fmt
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revise test logic for closing a follower region
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add sync region instruction for repartition procedure
This commit introduces a new sync region instruction and integrates it
into the repartition procedure flow, specifically for metric engine tables.
Changes:
- Add SyncRegion instruction type and SyncRegionsReply in instruction.rs
- Implement SyncRegionHandler in datanode to handle sync region requests
- Add SyncRegion state in repartition procedure to sync newly allocated regions
- Integrate sync region step after enter_staging_region for metric engine tables
- Add sync_region flag and allocated_region_ids to PersistentContext
- Make SyncRegionFromRequest serializable for instruction transmission
- Add test utilities and mock support for sync region operations
The sync region step is conditionally executed based on the table engine type,
ensuring that newly allocated regions in metric engine tables are properly
synced from their source regions before proceeding with manifest remapping.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: add logs
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(repartition): improve staging region handling and support metric engine repartition
- Reorder sync region flow: move SyncRegion from EnterStagingRegion to RepartitionStart to sync before applying staging
- Add ExitStaging metadata update state to properly clear staging leader info after repartition completes
- Update build_template_from_raw_table_info to optionally skip metric engine internal columns when creating region requests
- Fix region state transition: set_dropping now expects specific state (Staging or Writable) for proper validation
- Adjust region drop and copy handlers to handle staging regions correctly
- Add comprehensive test cases for metric engine SPLIT/MERGE partition operations on physical tables with logical tables
- Improve logging for table route updates, region drops, and repartition operations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: removes code duplication
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: update result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: refine comments
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: add error strategy support for flush region and flush pending deallocate regions
- **Add `ErrorStrategy` enum** in `procedure/utils.rs`:
- Supports `Ignore` and `Retry` strategies for error handling
- Refactor `flush_region` to accept `error_strategy` parameter
- Extract `handle_flush_region_reply` helper function for better code organization
- **Add pending deallocate region support**:
- Add `pending_deallocate_region_ids` field to `PersistentContext`
- Implement `flush_pending_deallocate_regions` in `EnterStagingRegion` state
- Flush pending deallocate regions before entering staging regions to ensure data consistency
- **Update error handling**:
- `flush_leader_region`: Use `ErrorStrategy::Ignore` to skip unreachable datanodes
- `sync_region`: Use `ErrorStrategy::Retry` for critical operations
- `enter_staging_region`: Use `ErrorStrategy::Retry` when flushing pending deallocate regions
This change improves the robustness of the repartition procedure by:
1. Providing flexible error handling strategies for flush operations
2. Ensuring pending deallocate regions are properly flushed before repartitioning
3. Preventing data inconsistency during region migration
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: compile
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: impl vector index scan in storage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: fallback to read remote blob when blob not found
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: refactor encoding and decoding and apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: license
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add apply_with_k tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: forgot to align nulls when the vector column is not in the batch
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add test for vector column is not in a batch while buiilding
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: implement deallocate regions for repartition procedure
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(metric-engine): add force flag to drop physical regions with associated logical regions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: update table metadata after deallocating regions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update proto
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: impl vector index building
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: supports flat format
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: add vector_index feature to test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions from copilot
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Change visibility of estimated_bytes, time_range, max_sequence, and
series_count fields from private to public for external access.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: implement per range stats for MemtableRange
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: extract methods to MemtableRanges
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: simple bulk memtable set other fields in stats
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: use time_index_type()
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: use time index type
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore/expose-symbols:
### Commit Message
Enhance `merge_and_dedup` Functionality in `flush.rs`
- **Function Signature Update**: Modified the `merge_and_dedup` function to accept `append_mode` and `merge_mode` as separate parameters instead of using `options`.
- **Function Accessibility**: Changed the visibility of `merge_and_dedup` to `pub` to allow external access.
- **Function Calls Update**: Updated calls to `merge_and_dedup` within `memtable_flat_sources` to align with the new function signature, passing `options.append_mode` and `options.merge_mode()` directly.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/expose-symbols:
### Add Merge and Deduplication Functionality
- **File**: `src/mito2/src/flush.rs`
- Introduced `merge_and_dedup` function to merge multiple record batch iterators and apply deduplication based on specified modes.
- Added detailed documentation for the function, explaining its arguments, behavior, and usage examples.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/expose-symbols:
## Refactor `bulk/part.rs` to Simplify Mutation Handling
- Removed the `mutations_to_record_batch` function and its associated helper functions, including `ArraysSorter`, `timestamp_array_to_iter`, and `binary_array_to_dictionary`, to simplify the mutation handling logic in `bulk/part.rs`.
- Deleted related test functions `check_binary_array_to_dictionary` and `check_mutations_to_record_batches` from the test module, along with their associated test cases.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/expose-symbols:
### Commit Message
**Refactor and Enhance Deduplication Logic**
- **`flush.rs`**: Refactored `maybe_dedup_one` function to accept `append_mode` and `merge_mode` as parameters instead of `RegionOptions`. This change enhances flexibility in deduplication logic.
- **`memtable/bulk.rs`**: Made `BulkRangeIterBuilder` struct and its fields public to allow external access and modification, improving extensibility.
- **`sst.rs`**: Corrected a typo in the schema documentation, changing `__prmary_key` to `__primary_key` for clarity and accuracy.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: collect per file metrics
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: divide build_cost to build_part_cost and build_reader_cost
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: limit the file metrics num to display
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: use sorted iter to get sorted files
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: output metrics in desc order
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
