* refactor(cli): unify storage configuration for export command
- Utilize ObjectStoreConfig to unify storage configuration for export command
- Support export command for Fs, S3, OSS, GCS and Azblob
- Fix the Display implementation for SecretString always returned the string
"SecretString([REDACTED])" even when the internal secret was empty.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Change the encapsulation permissions of each configuration
options for every storage backend to public access.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Update the implementation of ObjectStoreConfig::build_xxx() using macro solutions
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Introduce config validation for each storage type
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Enable trait-based polymorphism for storage type handling
(from inherent impl to trait impl)
- Extract helper functions to reduce code duplication
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Improve SecretString handling and validation
(Distinguishing between "not provided" and "empty string")
- Add validation when using filesystem storage
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Refactor storage field validation with macro
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- support GCS Application Default Credentials (like GKE, Cloud Run, or local development with ) in export
(Enabling ADC without validating or to be present)
(Making optional in GCS validation (defaults to https://storage.googleapis.com))
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
This commit refactors the validation logic for object store configurations in the CLI to leverage clap features and reduce boilerplate.
Key changes:
- Update wrap_with_clap_prefix macro to use clap's requires attribute.
This ensures that storage-specific options (e.g., --s3-bucket) are only accepted when the corresponding backend is enabled (e.g., --s3).
- Simplify FieldValidator trait by removing the is_provided method, as dependency checks are now handled by clap.
- Introduce validate_backend! macro to standardize the validation of required fields for enabled backends.
- Refactor ExportCommand to remove explicit validation calls (validate_s3, etc.) and rely on the validation within backend constructors.
- Add integration tests for ExportCommand to verify build success with S3, OSS, GCS, and Azblob configurations.
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
* refactor(cli): unify storage configuration for export command
- Use macros to simplify storage export implementation
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Rollback StorageExport trait implementation to not using macro for better code clarity and maintainability
- Introduce format_uri helper function to unify URI formatting logic
- Fix OSS URI path bug inherited from legacy code
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
* refactor(cli): unify storage configuration for export command
- Remove unnecessary async_trait
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
---------
Signed-off-by: McKnight22 <tao.wang.22@outlook.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
* chore: add tool to export db meta
* chore: add meta restore command
* chore: fmt code
* chore: remove useless error
* chore: support key prefix
* chore: add clean check for meta restore
* chore: add more log for meta restore
* chore: resolve s3 and local file root in command meta-snapshot
* chore: remove the pg mysql features from the build script as they are already in the default feature
* chore: fix by pr comment
* fix: drop unused dep using udeps to minial the size
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
* fix: adress comments fix the problem
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
---------
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
* feat: txn for pg kv backend
* chore: clippy
* fix: txn uses one client
* test: clean up and txn test
* test: clean up
* test: change lock_id to avoid conflict in test
* test: use different prefix in pg election test
* fix(test): just a fix
* test: aggregate multiple test to avoid concurrency problem
* test: use uuid instead of rng
* perf: batch cmp in txn
* perf: batch same op in txn
