mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-07-06 05:50:38 +00:00
3758911a2e
* feat: add close-first soft-drop table flow Soft-drop now tombstones table metadata and closes datanode regions instead of issuing physical drop requests, while preserving hard-drop cleanup semantics and blocking conflicting drops of recreated table names. Files: - `src/common/meta/src/ddl.rs` - `src/common/meta/src/ddl/drop_table.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/ddl_manager.rs` - `src/meta-srv/src/metasrv/builder.rs` - `src/cmd/src/standalone.rs` - `src/common/meta/src/test_util.rs` - `src/meta-srv/src/procedure/utils.rs` - `tests-integration/src/standalone.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add undrop and purge table procedures Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl_manager.rs\` - \`src/common/meta/src/rpc/ddl.rs\` - \`src/common/meta/src/key.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` - \`src/mito2/src/engine/open_test.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen soft-dropped regions before purge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable soft-drop operations for metric logical tables Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases. Files: - `src/common/meta/src/ddl/drop_table/metadata.rs` - `src/common/meta/src/ddl/purge_dropped_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/utils.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sync failure detectors during soft-drop lifecycle Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions. Files: - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/drop_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify soft-drop table tests Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused. Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/key.rs` - `src/mito2/src/engine/open_test.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen follower regions during undrop Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: use dropped table ddl expr protos Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: use inline dropped table task protos Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: various soft-drop improvements - Bump `greptime-proto` dependency revision (`Cargo.toml`, `Cargo.lock`) - Pass `region_wal_options` directly without pre-serialization in undrop flow (`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`) - Remove unused `RegionNumber` import (`src/common/meta/src/ddl/utils.rs`) - Add `reset_failure_detectors` to test mock (`src/common/meta/src/ddl/tests/drop_table.rs`) - Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask` (`src/common/meta/src/rpc/ddl.rs`) Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: validate table name match in undrop procedure Add a guard in `UndropTableProcedure::on_prepare()` to check that the dropped table name matches the undrop task name, returning `TableNotFound` on mismatch. This prevents undropping a table by a different name when only the table ID is known. - `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation - `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: simplify UndropTableTask to use table_id only Remove catalog, schema, and table name fields from `UndropTableTask` since the table name can be derived from the dropped table metadata in the procedure itself. This eliminates redundant fields and the associated name-validation test. Simplify locking in `UndropTableProcedure` to only use `TableLock`. Update `greptime-proto` dependency revision. - `Cargo.toml`, `Cargo.lock` - `src/common/meta/src/rpc/ddl.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: detect table name conflict during tombstone restore in undrop - Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore - Added `TombstoneTargetAlreadyExists` error variant - Map tombstone conflict to `TableAlreadyExists` in undrop procedure - Added test for undrop failing when live name created after prepare Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/key/tombstone.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: make PurgeDroppedTableTask table_id-only Streamline the purge-dropped-table flow by requiring a table_id instead of allowing name-based fallback. - Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs` - Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs` - Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs` - Bumped `greptime-proto` dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): chunk values by per-key txn ops instead of fixed divisor Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`) to correctly account for per-key transaction operations. Added `TxnOpLimitKvBackend` test helper and two new tests (`test_restore_chunks_by_total_txn_ops_limit`, `test_create_chunks_by_total_txn_ops_limit`) verifying chunking under tight txn op limits. Affected file: - `src/common/meta/src/key/tombstone.rs` — chunk size fix, `TxnOpLimitKvBackend` helper, two new tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge - `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before transitioning to DeleteTombstone state - `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into `open_regions_inner` with an `ignore_region_not_found` flag; expose `open_regions_ignore_region_not_found` for purge replayer - `src/common/meta/src/ddl/purge_dropped_table.rs`: use `open_regions_ignore_region_not_found` in replayed purge procedures - `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop idempotency and purge replay tolerance of dropped regions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: fix clippy Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(soft-drop): open regions before restoring undrop metadata Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): fail fast on invalid txn op budget Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump greptime-proto to main branch commit Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Setup tests for multiple storage backend
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
Take s3 for example. You need to set your S3 bucket, access key id and secret key:
# Settings for s3 test
GT_S3_BUCKET=S3 bucket
GT_S3_REGION=S3 region
GT_S3_ACCESS_KEY_ID=S3 access key id
GT_S3_ACCESS_KEY=S3 secret access key
Run
Execute the following command in the project root folder:
cargo test integration
Test s3 storage:
cargo test s3
Test oss storage:
cargo test oss
Test azblob storage:
cargo test azblob
Setup tests with Kafka wal
To run the integration test, please copy .env.example to .env in the project root folder and change the values on need.
GT_KAFKA_ENDPOINTS = localhost:9092
Setup kafka standalone
cd tests-integration/fixtures
docker compose -f docker-compose.yml up kafka
Setup tests with etcd TLS
This guide explains how to set up and test TLS-enabled etcd connections in GreptimeDB integration tests.
Quick Start
TLS certificates are already at tests-integration/fixtures/etcd-tls-certs/.
-
Start TLS-enabled etcd:
cd tests-integration/fixtures docker compose up etcd-tls -d -
Start all services (including etcd-tls):
cd tests-integration/fixtures docker compose up -d --wait
Certificate Details
The checked-in certificates include:
ca.crt- Certificate Authority certificateserver.crt/server-key.pem- Server certificate for etcd-tls serviceclient.crt/client-key.pem- Client certificate for connecting to etcd-tls
The server certificate includes SANs for localhost, etcd-tls, 127.0.0.1, and ::1.
Regenerating Certificates (Optional)
If you need to regenerate the etcd certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate-etcd-tls-certs.sh
# Or generate in custom location
./scripts/generate-etcd-tls-certs.sh /path/to/cert/directory
If you need to regenerate the mysql and postgres certificates:
# Regenerate certificates (overwrites existing ones)
./scripts/generate_certs.sh
# Or generate in custom location
./scripts/generate_certs.sh /path/to/cert/directory
Note: The checked-in certificates are for testing purposes only and should never be used in production.