Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): skip reopening dropped tables during purge
Purge soft-dropped tables by dropping stored routes directly instead of
reopening tombstoned regions first. Treat legacy
\`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only
transition to \`DropRegions\`.
Files:
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): guard undrop restore race
Serialize `UNDROP TABLE` with same-name creates by seeding the
original table name before procedure submission, and clean up reopened
regions when metadata restore fails.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(datanode): reject cleanup for registered regions
Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping.
Add coverage for `OfflineCleanup` engine selection and registered-region rejection.
Files:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): require tombstone before undrop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: harden soft-drop cleanup paths
Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop.
Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/mito2/src/engine/open_test.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump cleanup proto dependency
Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/store-api/src/region_request.rs\`
- \`src/common/meta/src/ddl/drop_table/executor.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: share region cleanup helpers
Share common region cleanup helpers between normal drop and offline cleanup
while keeping their preconditions separate.
- Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`.
- Share runtime artifact and manifest cache cleanup after full deletion paths.
- Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC.
Files:
- `src/mito2/src/worker/handle_drop.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: flush soft-dropped regions on close
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): handle flush-on-close race with concurrent in-flight flush
When a region close with `flush_on_close: true` races with an
already-running flush, pass the actual close request (including the
flush_on_close flag) to the DDL handler instead of a default request
so the pending flush is correctly awaited.
Files: `src/mito2/src/worker/handle_close.rs`
Also adds a test verifying that closing with flush-on-close while a
flush is in progress still persists all written data correctly.
Files: `src/mito2/src/engine/close_test.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: finish close request callsites
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: replace imperative close-region loop with functional combinators
Transform the region close dispatch in `DropTableExecutor` from mutable
`Vec` and `push` loops to iterator chains with `join_all`, improving
idiomatic Rust style and readability.
- `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode
region-close logic to use `peers.map()` and nested `join_all`, moving
`node_manager.datanode()` inside the closure to align with the new
structure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: decouple Kafka client from WAL checkpoint
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: merge Kafka WAL index checkpoints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: delegate Kafka WAL retirement to metasrv
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: rebase main and resolve conflicts
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: license header
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove Kafka obsolete-all index changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add undrop and purge table procedures
Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen soft-dropped regions before purge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable soft-drop operations for metric logical tables
Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.
Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sync failure detectors during soft-drop lifecycle
Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.
Files:
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/drop_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify soft-drop table tests
Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/key.rs`
- `src/mito2/src/engine/open_test.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen follower regions during undrop
Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: use dropped table ddl expr protos
Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: use inline dropped table task protos
Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: various soft-drop improvements
- Bump `greptime-proto` dependency revision
(`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
(`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
(`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
(`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
(`src/common/meta/src/rpc/ddl.rs`)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: validate table name match in undrop procedure
Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch. This prevents undropping a table by a different name when
only the table ID is known.
- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: simplify UndropTableTask to use table_id only
Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.
Simplify locking in `UndropTableProcedure` to only use `TableLock`.
Update `greptime-proto` dependency revision.
- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: detect table name conflict during tombstone restore in undrop
- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore
- Added `TombstoneTargetAlreadyExists` error variant
- Map tombstone conflict to `TableAlreadyExists` in undrop procedure
- Added test for undrop failing when live name created after prepare
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/key/tombstone.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: make PurgeDroppedTableTask table_id-only
Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.
- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor
Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.
Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
`TxnOpLimitKvBackend` helper, two new tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge
- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
`open_regions_inner` with an `ignore_region_not_found` flag; expose
`open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
`open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
idempotency and purge replay tolerance of dropped regions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: fix clippy
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(soft-drop): open regions before restoring undrop metadata
Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): fail fast on invalid txn op budget
Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to main branch commit
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record catalog and schema in slow queries
Add catalog and schema context to slow query records while appending the new columns after existing fields to preserve column order.
- `src/common/frontend/src/slow_query_event.rs`: extend `SlowQueryEvent` schema and rows with `catalog_name` and `schema_name`, and cover append-only ordering.
- `src/catalog/src/process_manager.rs`: carry catalog and schema through `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: capture context for SQL, plan, and PromQL slow query timers.
- `tests-integration/tests/sql.rs`: assert MySQL and PostgreSQL slow query records include catalog and schema.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: address slow query review comment
Use `String::clone` when writing slow query catalog and schema values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep slow query schema only
Remove the slow query `catalog_name` column and keep `schema_name` as a non-null tag dimension.
- `src/common/frontend/src/slow_query_event.rs`: expose only `schema_name` in `SlowQueryEvent` rows and mark it as a tag.
- `src/catalog/src/process_manager.rs`: stop carrying catalog context in `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: pass only schema context to slow query timers.
- `tests-integration/tests/sql.rs`: assert slow query records include `schema_name` without `catalog_name`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: schema name semantic should be field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: declare GreptimeDB Enterprise License for enterprise-gated sources
The `enterprise`-feature-gated sources (triggers, mito2 extension) were
excluded from the Apache-2.0 header check but carried no license of their
own. Declare a separate GreptimeDB Enterprise License and enforce it.
- Add LICENSE-ENTERPRISE (open-core split; core stays Apache-2.0).
- Add an Enterprise License header to each enterprise source file.
- Add licenserc-enterprise.toml and a second hawkeye step in CI to enforce
the Enterprise header on exactly those files.
- Cross-reference the two complementary file lists; document the layout in
licenses/README.md and the README License section.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: reference per-customer Enterprise Agreement instead of a terms URL
There is no public enterprise-terms page; each customer signs an individually
negotiated agreement. Point the license at a "separate written commercial
agreement with GrepTime Inc." and direct readers to the existing contact page.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support defer_on_missing_source for pending flow creation
Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.
Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
runtime/flownode `CreateRequest` but preserved in metadata for
SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
missing and `defer_on_missing_source=true`; falls back to
`FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
activation time
- Validated flow options: only `defer_on_missing_source` allowed;
unknown options rejected
- Known issue: standalone mode does not support flownodes, so
pending flow flush/sink behavior covered only in distributed
sqlness; operator and meta unit tests cover activation logic
Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip
Signed-off-by: discord9 <discord9@163.com>
* chore: simplify pending flow PR scope
Reduce PR #8124 to the metadata-only MVP after complexity review.
Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only
Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.
Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`
Signed-off-by: discord9 <discord9@163.com>
* test: cover pending flow metadata edge cases
Signed-off-by: discord9 <discord9@163.com>
* test: fix pending flow metadata test lint
Signed-off-by: discord9 <discord9@163.com>
* docs: document pending flow metadata fields
Signed-off-by: discord9 <discord9@163.com>
* chore: more sleep when test
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* feat: expose node info for placement selectors
Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.
Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.
Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip datanode node info without stats
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: avoid unnecessary workload clones
Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.
Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: require frontend origin address
Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.
Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: reuse resolved frontend address
Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.
Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve datanode lease liveness
Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.
Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Remove stale datanode lease helper
- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/env-based-region-selector-oss: simplify lease discovery
- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: support reporting env vars in heartbeat messages to metasrv
Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).
- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
`common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
`FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
`CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
`discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: address heartbeat env review feedback
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: log error on deserialization failure
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: send heartbeat env vars once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: resend heartbeat env vars after reconnect
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: keep env vars in every heartbeat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Weny Xu
jeremyhi
raphaelroshan
shuiyisong
Lei, HUANG
discord9
dennis zhuang
QuakeWang
