* fix: allow ipv4_num_to_string to accept valid integers
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
* test: update sqlness result file
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
* fix: use coercible integer signature for ipv4_num_to_string
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
---------
Signed-off-by: Johannes Sluis <joesluis51@gmail.com>
* feat(mito2): extract and cache primary key range for SST files
Extracts primary key ranges from SST files during flush and compaction, and caches them in FileHandle for future use (e.g., overlapping checks).
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/compaction-overlapping-check:
- **Enhance Primary Key Range Logic**: Updated the `primary_key_ranges_overlap` function in `run.rs` to use `chunk()` for comparing byte ranges, improving accuracy in overlap detection.
- **Refactor Run Assignment Logic**: Simplified the logic for assigning items to runs in `run.rs` by removing redundant match statements and using `is_empty()` and `iter().any()` for cleaner checks.
- **Add Test for Transitivity Break**: Introduced a new test `test_find_sorted_runs_handles_2d_transitivity_break` in `run.rs` to ensure correct handling of transitivity breaks in sorted runs.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/compaction-overlapping-check:
- **Remove PK-disjoint detection logic**: Simplified the compaction logic in `twcs.rs` by removing the `has_time_overlapping_pairs` function and related logic for PK-disjoint detection. This includes the removal of the `append_mode_force_compact` condition and associated tests.
- **Update compaction trigger settings**: Modified `append_mode_test.rs` to set `compaction.twcs.trigger_file_num` to "2" and adjusted the expected number of files in the scanner assertion from 1 to 2.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: rebase main
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/compaction-overlapping-check:
### Remove Unused Import in `compactor.rs`
- Removed the unused import `compact_request` from `compactor.rs` to clean up the codebase.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: tighten `mito2` file lifecycle handling
Refine compaction, flush, and SST/version bookkeeping across `src/mito2/src/compaction/*`, `src/mito2/src/flush.rs`, `src/mito2/src/region/*`, `src/mito2/src/sst/*`, and related tests/utilities.
* fix: reuse primary key range merge in twcs compaction
Centralize primary key range merging so can call the shared helper from .
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor(mito2): simplify `FileHandle` initialization and internalize primary key range extraction
- Updated `FileHandle::new` to automatically compute the primary key range directly from `FileMeta`.
- Restricted `FileHandle::new_with_primary_key_range` to be test-only by adding the `#[cfg(test)]` attribute.
- Simplified `SstVersion::add_files` by adopting the updated `FileHandle::new` instead of manually providing the primary key range.
Modified files:
- `src/mito2/src/sst/file.rs`
- `src/mito2/src/sst/version.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/compaction-overlapping-check:
### Improve File Management and Documentation
- **`twcs.rs`**: Added a comment to clarify the merging of small files when there are no overlapping files.
- **`version.rs` (in `region` and `sst`)**: Enhanced documentation for `add_files` method, explaining its functionality and panic conditions. Simplified the file handle creation logic in `sst/version.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/compaction-overlapping-check:
### Enhance Primary Key Range Handling in `opener.rs`
- Updated logic in `opener.rs` to set the primary key range for file handles when it is not already defined. This change ensures that the primary key range is extracted and set using `extract_primary_key_range` when necessary.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* docs: add docs for the necessity of checking pk and timesmaps while find overlapping files.
* chore: address review comments
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor(meta-srv): move group rollback ownership to parent repartition procedure
- Parent procedure now owns partial rollback based on failed/unknown subprocedures
- rollback order: group metadata first, then allocated-region cleanup
- original_target_routes captured during build-plan, persisted in RepartitionPlanEntry
- rollback_group_metadata_routes moved to utils as parent-owned helper
- Group subprocedure no longer supports rollback (rollback_supported = false)
- Removed UpdateMetadata::RollbackStaging from group state machine
- Deleted redundant group rollback tests and helpers
BREAKING CHANGE: group Procedure no longer handles rollback; parent procedure
is responsible for crash recovery and selecting which plans to roll back.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update comments
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
refactor/remove-compactor-compact:
### Remove Unused Compaction Functionality
- **Removed `compact` Method**: Eliminated the `compact` method from the `Compactor` trait and its default implementation, which was primarily used for local compaction in testing. This change affects `compactor.rs`.
- **Code Cleanup**: Removed associated code and comments related to the `compact` method, streamlining the `Compactor` trait interface.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: fix git cliff errors in latest version
- Fix errors in v2.12.0
- Do not generate logs for beta/rc tags between the compared commits
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: preserve blank line before release date in changelog
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): improve compaction error handling and file removal
Refactor compaction task execution to enhance error handling and robustness.
- Implemented parallel execution of compaction tasks with proper error capture and logging for individual task failures.
- Ensured JoinSnafu is no longer directly used in error propagation, instead handling errors within the task processing loop.
- Adjusted file removal logic to correctly include expired SSTs after compaction merges.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor(mito2): extract SstMerger trait for testability in compaction
Extract SstMerger trait and DefaultSstMerger implementation to improve the testability of DefaultCompactor.
The DefaultCompactor is now generic over SstMerger, allowing mock implementations to be injected for unit testing without relying on the full object storage access layer. This refactoring separates the concerns of SST file merging from the overall compaction orchestration logic.
Additionally:
- Updated CompactionScheduler to use DefaultCompactor::default().
- Added unit tests for DefaultCompactor using a MockMerger.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(compaction): propagate join error during sst flush
Correctly propagates the error when joining SST flush handles during compaction. Previously, the error was logged but not returned, leading to potential silent failures.
Also reorders some imports for consistency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(compaction): pre-allocate capacity for compacted_inputs
Pre-allocates capacity for the compacted_inputs vector based on the estimated total size of inputs and expired SSTs. This optimization aims to reduce vector reallocations during the compaction process.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-partial-compaction:
### Commit Message
Enhance `DefaultCompactor` and `MockMerger` for Improved Flexibility
- **`compactor.rs`**:
- Added `Clone` trait to `DefaultSstMerger` and `MockMerger` to allow cloning.
- Removed `Arc` wrapping from `DefaultCompactor`'s `merger` field for direct usage.
- Updated `merge_ssts` method to require `Clone` trait for `SstMerger`.
- Modified `MockMerger` to use `Arc<Mutex>` for `results` and `call_idx` to ensure thread safety.
- Adjusted error handling to use `error::InvalidMetaSnafu` directly.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
fix/flaky-test:
### Add Dynamic Port Selection for Standalone Tests
- **`cli.rs`**: Implemented functions `random_standalone_addrs` and `choose_random_unused_port_offset` to dynamically select unused ports for standalone tests, enhancing test reliability.
- Updated `test_export_create_table_with_quoted_names` to use dynamically assigned ports for HTTP, RPC, MySQL, and PostgreSQL addresses.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(index): intersect bitmaps before early exit in predicates applier
The loop skipped intersecting when the next bitmap was empty, which left
the accumulator unchanged instead of zeroing it. Intersect first, then
break when the result is empty.
Signed-off-by: Weixie Cui <cuiweixie@gmail.com>
* per gemini
* style(index): format predicates applier loop
* fix(index): remove unused mut in predicates applier
---------
Signed-off-by: Weixie Cui <cuiweixie@gmail.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
* feat: add support for decimal parameter type, remove string replacement fallback
* chore: format
* fix: add support for using unsigned bigint in postgres
* chore: format toml
* refactor: cleanup duplicated code
* fix: rescale decimal
perf: move Tantivy fulltext search to blocking thread pool
Wrap the synchronous Tantivy search (query parsing, posting list
traversal, stored field reads) in spawn_blocking_global to avoid
starving the tokio async runtime with CPU-bound work.
Signed-off-by: lyang24 <lanqingy93@gmail.com>
