- Performance: add collect_lightweight_query_load_metrics to walk the
physical plan and read raw metric values without invoking MetricCollector's
plan-node formatting on the normal query hot path before EOF.
- Refactor: extract collect_full_metrics and keep full aggregation/formatting
for verbose analyze output and terminal metrics.
- Test: cover lightweight partial metrics and drop-time query stats reuse.
Files: src/common/recordbatch/src/adapter.rs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Ensure read requests from `RegionServer` are dispatched to the query
runtime via `common_runtime::spawn_query` instead of executing on the
calling thread. This prevents runtime contention and keeps read
processing on the designated query runtime.
Affected file:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add undrop and purge table procedures
Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen soft-dropped regions before purge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable soft-drop operations for metric logical tables
Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.
Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sync failure detectors during soft-drop lifecycle
Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.
Files:
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/drop_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify soft-drop table tests
Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/key.rs`
- `src/mito2/src/engine/open_test.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen follower regions during undrop
Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: use dropped table ddl expr protos
Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: use inline dropped table task protos
Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: various soft-drop improvements
- Bump `greptime-proto` dependency revision
(`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
(`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
(`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
(`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
(`src/common/meta/src/rpc/ddl.rs`)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: validate table name match in undrop procedure
Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch. This prevents undropping a table by a different name when
only the table ID is known.
- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: simplify UndropTableTask to use table_id only
Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.
Simplify locking in `UndropTableProcedure` to only use `TableLock`.
Update `greptime-proto` dependency revision.
- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: detect table name conflict during tombstone restore in undrop
- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore
- Added `TombstoneTargetAlreadyExists` error variant
- Map tombstone conflict to `TableAlreadyExists` in undrop procedure
- Added test for undrop failing when live name created after prepare
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/key/tombstone.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: make PurgeDroppedTableTask table_id-only
Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.
- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor
Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.
Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
`TxnOpLimitKvBackend` helper, two new tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge
- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
`open_regions_inner` with an `ignore_region_not_found` flag; expose
`open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
`open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
idempotency and purge replay tolerance of dropped regions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: fix clippy
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(soft-drop): open regions before restoring undrop metadata
Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): fail fast on invalid txn op budget
Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to main branch commit
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record catalog and schema in slow queries
Add catalog and schema context to slow query records while appending the new columns after existing fields to preserve column order.
- `src/common/frontend/src/slow_query_event.rs`: extend `SlowQueryEvent` schema and rows with `catalog_name` and `schema_name`, and cover append-only ordering.
- `src/catalog/src/process_manager.rs`: carry catalog and schema through `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: capture context for SQL, plan, and PromQL slow query timers.
- `tests-integration/tests/sql.rs`: assert MySQL and PostgreSQL slow query records include catalog and schema.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: address slow query review comment
Use `String::clone` when writing slow query catalog and schema values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep slow query schema only
Remove the slow query `catalog_name` column and keep `schema_name` as a non-null tag dimension.
- `src/common/frontend/src/slow_query_event.rs`: expose only `schema_name` in `SlowQueryEvent` rows and mark it as a tag.
- `src/catalog/src/process_manager.rs`: stop carrying catalog context in `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: pass only schema context to slow query timers.
- `tests-integration/tests/sql.rs`: assert slow query records include `schema_name` without `catalog_name`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: schema name semantic should be field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: disable non object json write
* remove unused code
* fix: sqlness test
* simply code
* add comment
* remove unused unit test
* fix: cargo fmt
* fix: unit test
* fix: unit test
* refactor: extract region hook function for extension
* refactor: add a warning for unmatched vector size
* feat: reconstruct sst_info from file_meta
* chore: update comments about the hook
Make auto_flush_interval configurable per table (region) in addition to
the global Mito engine config, mirroring how ttl supports a global default
with a per-table override.
- Add auto_flush_interval to RegionOptions, parsed from the table option
with the same humantime format as the global config; reject a
non-positive value.
- The periodic flush logic resolves the effective interval per region,
falling back to the global config when unset, using a saturating
conversion to avoid overflow on extreme values.
- Accept the option key in is_mito_engine_option_key so it can be set at
CREATE TABLE via WITH ('auto_flush_interval' = '5m').
Ref #8340
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* feat(json2): type hint
* test(datatypes): add JsonSettings serde round-trip test
* minor refactor
This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.
* fix: code review
* feat: accept x-greptime-pipeline-name header on /events/logs
The /events/logs (and /logs/ingest) endpoint previously only read the
pipeline name from the `pipeline_name` query parameter, while the
OTLP/Elasticsearch/Splunk log ingestion endpoints already accept it via
the `x-greptime-pipeline-name` header. This inconsistency is unfriendly
to users.
Make `log_ingester` resolve the pipeline name from the
`x-greptime-pipeline-name` header (and the deprecated
`x-greptime-log-pipeline-name`), falling back to the query parameter.
The header takes precedence, consistent with how other pipeline options
(e.g. `x-greptime-pipeline-params`) outrank their query-parameter
counterparts.
Closes#6095
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* address review: prefer non-deprecated pipeline-name header
When both pipeline-name headers are present, resolve the non-deprecated
`x-greptime-pipeline-name` before the deprecated
`x-greptime-log-pipeline-name`, and cover the precedence with a test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
