mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-01-05 12:52:57 +00:00
6c57f4b7e4
* feat: persist column ids in table metadata (#6457) * feat: persist column ids in table metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add column metadata to response extensions (#6451) Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): extract `AlterTableExecutor` from `AlterTableProcedure` (#6470) * refactor(meta): extract `AlterTableExecutor` from `AlterTableProcedure` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): separate validation and execution logic in alter logical tables procedure (#6478) * refactor(meta): separate validation and execution logic in alter logical tables procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix state transition in create table procedure (#6523) Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add table reconciliation utilities (#6519) * feat: add table reconciliation utilities Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestison from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update comment Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: Support ListMetadataRequest to retrieve regions' metadata (#6348) * feat: support list metadata in region server Signed-off-by: evenyag <realevenyag@gmail.com> * test: add test for list region metadata Signed-off-by: evenyag <realevenyag@gmail.com> * feat: return null if region not exists Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: support multiple index operations in single alter region request (#6487) * refactor: support multiple index operations in single alter region request Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: implement pause/resume functionality for procedure manager (#6393) * feat: implement pause/resume functionality for procedure manager Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: move metasrv admin to http server while keep tonic for backward compatibility (#6466) * feat: move metasrv admin to http server while keep tonic for backward compatibility Signed-off-by: lyang24 <lanqingy93@gmail.com> * refactor with nest method Signed-off-by: lyang24 <lanqingy93@gmail.com> --------- Signed-off-by: lyang24 <lanqingy93@gmail.com> Co-authored-by: lyang24 <lanqingy@usc.edu> * feat: allow igoring nonexistent regions in recovery mode (#6592) * feat: allow ignoring nonexistent regions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: ignore nonexistent regions during startup in recovery mode Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: allow enabling recovery mode via http api Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: allow setting next table id via http api (#6597) * feat: allow reset next table id via http api Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggesions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: ignore internal keys in metadata snapshots (#6606) feat: ignore dumpping internal keys Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile table procedure (#6584) * feat: introduce `SyncColumns` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile table procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggesions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile database procedure (#6612) * feat: introduce reconcile database procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: hold the schema lock Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add todo Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename to `fast_fail` Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile logical tables procedure (#6588) * feat: introduce reconcile logical tables procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: lock logical tables Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove procedure executor from DDL manager (#6625) * refactor: remove procedure executor from DDL manager Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile catalog procedure (#6613) Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconciliation interface (#6614) * feat: introduce reconcile interface Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: upgrade proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix sequence peek method to return correct values when sequence is not initialized (#6643) fix: improve sequence peek method to handle uninitialized sequences Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: sequence peek with remote value (#6648) * fix: sequence peek with remote value * chore: more ut * chore: add more ut * feat: add metrics for reconciliation procedures (#6652) * feat: add metrics for reconciliation procedures Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: improve error handling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(datanode): handle ignore_nonexistent_region flag in open_all_regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: merge metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): add metadata region cache (#6657) * feat(metric-engine): add metadata region cache Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: use lru Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: rename Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: default ttl Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: longer ttl Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump version to 0.15.5 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: lyang24 <lanqingy93@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: Lanqing Yang <lanqingy93@gmail.com> Co-authored-by: lyang24 <lanqingy@usc.edu> Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose-standalone.yml up kafka