* feat(native-histogram): store counts and span lengths as signed integers
Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:
- count_u64 / zero_count_u64: uint64 -> int64
- positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
- Span.length (query-time model): u32 -> i32
The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.
The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.
Signed-off-by: Ning Sun <sunning@greptime.com>
* refactor(native-histogram): rename count/zero_count fields to _i64
Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:
COUNT_U64_FIELD ("count_u64") -> COUNT_I64_FIELD ("count_i64")
ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")
The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).
Signed-off-by: Ning Sun <sunning@greptime.com>
* test(native-histogram): refresh planner plan snapshot for signed types
The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:
positive/negative_span_lengths: List(UInt32) -> List(Int32)
count_u64/zero_count_u64: UInt64 -> count_i64/zero_count_i64: Int64
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
(cherry picked from commit 3510ef7d4c)
(cherry picked from commit d636d6fd9e9474b262982f54dc5805f80127087a)
* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add a dedicated http api server port
* fix: integration test
* refactor: make http-api-port opt-in
* refactor: rename attribute to http-api-server
* feat: use middleware to check different http server port
* refactor: rename config option
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report.
Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations.
Signed-off-by: discord9 <discord9@163.com>
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add undrop and purge table procedures
Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen soft-dropped regions before purge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable soft-drop operations for metric logical tables
Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.
Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sync failure detectors during soft-drop lifecycle
Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.
Files:
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/drop_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify soft-drop table tests
Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/key.rs`
- `src/mito2/src/engine/open_test.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen follower regions during undrop
Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: use dropped table ddl expr protos
Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: use inline dropped table task protos
Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: various soft-drop improvements
- Bump `greptime-proto` dependency revision
(`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
(`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
(`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
(`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
(`src/common/meta/src/rpc/ddl.rs`)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: validate table name match in undrop procedure
Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch. This prevents undropping a table by a different name when
only the table ID is known.
- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: simplify UndropTableTask to use table_id only
Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.
Simplify locking in `UndropTableProcedure` to only use `TableLock`.
Update `greptime-proto` dependency revision.
- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: detect table name conflict during tombstone restore in undrop
- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore
- Added `TombstoneTargetAlreadyExists` error variant
- Map tombstone conflict to `TableAlreadyExists` in undrop procedure
- Added test for undrop failing when live name created after prepare
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/key/tombstone.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: make PurgeDroppedTableTask table_id-only
Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.
- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor
Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.
Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
`TxnOpLimitKvBackend` helper, two new tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge
- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
`open_regions_inner` with an `ignore_region_not_found` flag; expose
`open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
`open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
idempotency and purge replay tolerance of dropped regions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: fix clippy
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(soft-drop): open regions before restoring undrop metadata
Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): fail fast on invalid txn op budget
Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to main branch commit
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: record catalog and schema in slow queries
Add catalog and schema context to slow query records while appending the new columns after existing fields to preserve column order.
- `src/common/frontend/src/slow_query_event.rs`: extend `SlowQueryEvent` schema and rows with `catalog_name` and `schema_name`, and cover append-only ordering.
- `src/catalog/src/process_manager.rs`: carry catalog and schema through `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: capture context for SQL, plan, and PromQL slow query timers.
- `tests-integration/tests/sql.rs`: assert MySQL and PostgreSQL slow query records include catalog and schema.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: address slow query review comment
Use `String::clone` when writing slow query catalog and schema values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep slow query schema only
Remove the slow query `catalog_name` column and keep `schema_name` as a non-null tag dimension.
- `src/common/frontend/src/slow_query_event.rs`: expose only `schema_name` in `SlowQueryEvent` rows and mark it as a tag.
- `src/catalog/src/process_manager.rs`: stop carrying catalog context in `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: pass only schema context to slow query timers.
- `tests-integration/tests/sql.rs`: assert slow query records include `schema_name` without `catalog_name`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: schema name semantic should be field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Weny Xu
Ning Sun
Yingwen
Lei, HUANG
jeremyhi
discord9
shuiyisong
yimeng
Yash Agrawal
QuakeWang
fys