mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-01-14 01:02:55 +00:00
f3e2d333e4
* refactor: rename WalOptionsAllocator to WalProvider The name "WalOptionsAllocator" was misleading because: - For RaftEngine variant, it doesn't actually allocate anything - The actual allocation logic lives in KafkaTopicPool "WalProvider" better describes its role as providing WAL options based on the configured WAL backend (RaftEngine or Kafka). Changes: - Rename `WalOptionsAllocator` to `WalProvider` - Rename `WalOptionsAllocatorRef` to `WalProviderRef` - Rename `build_wal_options_allocator` to `build_wal_provider` - Rename module `wal_options_allocator` to `wal_provider` - Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`, `StartWalOptionsAllocator` -> `StartWalProvider` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): extract allocator traits from TableMetadataAllocator Refactor TableMetadataAllocator to use trait-based dependency injection for better testability and separation of concerns. Changes: - Add `ResourceIdAllocator` trait to abstract ID allocation - Add `WalOptionsAllocator` trait to abstract WAL options allocation - Implement traits for `Sequence` and `WalProvider` - Remove duplicate `allocate_region_wal_options` function - Rename `table_id_sequence` to `table_id_allocator` for consistency - Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): add max_region_number tracking to PhysicalTableRouteValue Add `max_region_number` field to track the highest region number ever allocated for a table. This value only increases when regions are added and never decreases when regions are dropped, ensuring unique region numbers across the table's lifetime. Changes: - Add `max_region_number` field to `PhysicalTableRouteValue` - Implement custom `Deserialize` for backward compatibility - Update `update_region_routes` to maintain max_region_number - Calculate max_region_number from region_routes in `new()` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: extract TableRouteAllocator trait from TableMetadataAllocator - Add TableRouteAllocator trait for abstracting region route allocation - Implement blanket impl for all PeerAllocator types - Add PeerAllocator impl for Arc<T> to support trait object delegation - Update TableMetadataAllocator to use TableRouteAllocatorRef Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename TableRouteAllocator to RegionRoutesAllocator - Rename table_route.rs to region_routes.rs - Rename TableRouteAllocator trait to RegionRoutesAllocator - Rename wal_option.rs to wal_options.rs for consistency - Update TableMetadataAllocator to use new naming Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta-srv): implement region allocation for repartition procedure This commit implements the region allocation phase of the repartition procedure, which handles allocating new regions when a table needs to be split into more partitions. Key changes: - Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples for more flexible region number assignment - Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate` fields (now derived from source/target counts) - Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal, and deallocation cases - Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly returning source regions) - Implement complete `AllocateRegion` state with: - Region route allocation via `RegionRoutesAllocator` - WAL options allocation via `WalOptionsAllocator` - Operating region registration for concurrency control - Region creation on datanodes via `CreateTableExecutor` - Table route metadata update - Add `TableRouteValue::max_region_number()` helper method - Add comprehensive unit tests for plan conversion and allocation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.