* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* refactor: rename WalOptionsAllocator to WalProvider
The name "WalOptionsAllocator" was misleading because:
- For RaftEngine variant, it doesn't actually allocate anything
- The actual allocation logic lives in KafkaTopicPool
"WalProvider" better describes its role as providing WAL options
based on the configured WAL backend (RaftEngine or Kafka).
Changes:
- Rename `WalOptionsAllocator` to `WalProvider`
- Rename `WalOptionsAllocatorRef` to `WalProviderRef`
- Rename `build_wal_options_allocator` to `build_wal_provider`
- Rename module `wal_options_allocator` to `wal_provider`
- Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`,
`StartWalOptionsAllocator` -> `StartWalProvider`
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(meta): extract allocator traits from TableMetadataAllocator
Refactor TableMetadataAllocator to use trait-based dependency injection
for better testability and separation of concerns.
Changes:
- Add `ResourceIdAllocator` trait to abstract ID allocation
- Add `WalOptionsAllocator` trait to abstract WAL options allocation
- Implement traits for `Sequence` and `WalProvider`
- Remove duplicate `allocate_region_wal_options` function
- Rename `table_id_sequence` to `table_id_allocator` for consistency
- Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler`
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(meta): add max_region_number tracking to PhysicalTableRouteValue
Add `max_region_number` field to track the highest region number ever
allocated for a table. This value only increases when regions are added
and never decreases when regions are dropped, ensuring unique region
numbers across the table's lifetime.
Changes:
- Add `max_region_number` field to `PhysicalTableRouteValue`
- Implement custom `Deserialize` for backward compatibility
- Update `update_region_routes` to maintain max_region_number
- Calculate max_region_number from region_routes in `new()`
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: extract TableRouteAllocator trait from TableMetadataAllocator
- Add TableRouteAllocator trait for abstracting region route allocation
- Implement blanket impl for all PeerAllocator types
- Add PeerAllocator impl for Arc<T> to support trait object delegation
- Update TableMetadataAllocator to use TableRouteAllocatorRef
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor: rename TableRouteAllocator to RegionRoutesAllocator
- Rename table_route.rs to region_routes.rs
- Rename TableRouteAllocator trait to RegionRoutesAllocator
- Rename wal_option.rs to wal_options.rs for consistency
- Update TableMetadataAllocator to use new naming
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat(meta-srv): implement region allocation for repartition procedure
This commit implements the region allocation phase of the repartition procedure,
which handles allocating new regions when a table needs to be split into more partitions.
Key changes:
- Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples
for more flexible region number assignment
- Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate`
fields (now derived from source/target counts)
- Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal,
and deallocation cases
- Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly
returning source regions)
- Implement complete `AllocateRegion` state with:
- Region route allocation via `RegionRoutesAllocator`
- WAL options allocation via `WalOptionsAllocator`
- Operating region registration for concurrency control
- Region creation on datanodes via `CreateTableExecutor`
- Table route metadata update
- Add `TableRouteValue::max_region_number()` helper method
- Add comprehensive unit tests for plan conversion and allocation logic
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Utilize ObjectStoreConfig to unify storage configuration for export command
- Support export command for Fs, S3, OSS, GCS and Azblob
- Fix the Display implementation for SecretString always returned the string
"SecretString([REDACTED])" even when the internal secret was empty.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Change the encapsulation permissions of each configuration
options for every storage backend to public access.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Update the implementation of ObjectStoreConfig::build_xxx() using macro solutions
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Introduce config validation for each storage type
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Enable trait-based polymorphism for storage type handling
(from inherent impl to trait impl)
- Extract helper functions to reduce code duplication
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Improve SecretString handling and validation
(Distinguishing between "not provided" and "empty string")
- Add validation when using filesystem storage
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Refactor storage field validation with macro
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- support GCS Application Default Credentials (like GKE, Cloud Run, or local development with ) in export
(Enabling ADC without validating or to be present)
(Making optional in GCS validation (defaults to https://storage.googleapis.com))
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
This commit refactors the validation logic for object store configurations in the CLI to leverage clap features and reduce boilerplate.
Key changes:
- Update wrap_with_clap_prefix macro to use clap's requires attribute.
This ensures that storage-specific options (e.g., --s3-bucket) are only accepted when the corresponding backend is enabled (e.g., --s3).
- Simplify FieldValidator trait by removing the is_provided method, as dependency checks are now handled by clap.
- Introduce validate_backend! macro to standardize the validation of required fields for enabled backends.
- Refactor ExportCommand to remove explicit validation calls (validate_s3, etc.) and rely on the validation within backend constructors.
- Add integration tests for ExportCommand to verify build success with S3, OSS, GCS, and Azblob configurations.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Use macros to simplify storage export implementation
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Rollback StorageExport trait implementation to not using macro for better code clarity and maintainability
- Introduce format_uri helper function to unify URI formatting logic
- Fix OSS URI path bug inherited from legacy code
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Remove unnecessary async_trait
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
---------
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
jeremyhi
Weny Xu
shuiyisong
Ning Sun
dependabot[bot]andNing Sun
LFC
fys
Lanqing Yang
