* fix(flow): convert streaming expiration to milliseconds
Flow EXPIRE AFTER values are stored in seconds, while streaming flow timestamps and durations use milliseconds. Passing the value through unchanged expires state and limits refill scans 1000 times too early.
Convert the value at streaming create and refill boundaries, reject overflow, and document the CreateFlowArgs unit.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* fix(flow): reject negative streaming expiration
Reject negative EXPIRE AFTER values at the streaming adapter boundary so refill ranges cannot move into the future. Keep zero valid and retain checked seconds-to-milliseconds conversion.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
---------
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* fix: flush soft-dropped regions on close
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): handle flush-on-close race with concurrent in-flight flush
When a region close with `flush_on_close: true` races with an
already-running flush, pass the actual close request (including the
flush_on_close flag) to the DDL handler instead of a default request
so the pending flush is correctly awaited.
Files: `src/mito2/src/worker/handle_close.rs`
Also adds a test verifying that closing with flush-on-close while a
flush is in progress still persists all written data correctly.
Files: `src/mito2/src/engine/close_test.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: finish close request callsites
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: replace imperative close-region loop with functional combinators
Transform the region close dispatch in `DropTableExecutor` from mutable
`Vec` and `push` loops to iterator chains with `join_all`, improving
idiomatic Rust style and readability.
- `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode
region-close logic to use `peers.map()` and nested `join_all`, moving
`node_manager.datanode()` inside the closure to align with the new
structure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: decouple Kafka client from WAL checkpoint
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: merge Kafka WAL index checkpoints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: delegate Kafka WAL retirement to metasrv
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: rebase main and resolve conflicts
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: license header
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove Kafka obsolete-all index changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Only fire `on_manifest_updated` for writes to the live (normal) manifest
directory. Writes to the staging directory (enter staging, operations during
staging, the intermediate apply-staging `RegionEdit`) are suppressed — their
effects accumulate and are delivered in a single notification when
`exit_staging_on_success` promotes all staged actions to the live manifest.
This simplifies hook implementations that only care about "live" state, since
they no longer receive intermediate staging notifications they cannot act on,
and no file-list information is lost (the promote step carries all accumulated
files from staging operations).
Key changes:
- `PendingManifestHook` gains an `is_staging` field; `fire()` is a no-op when
`is_staging` is `true`
- `ManifestContext::update_locked` threads its `is_staging` parameter through
- Updated module-level docs and coverage table
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: hook extension for region/file close/drop
* feat: cover region lifecycle with open/close/drop hooks
* docs(mito2): clarify on_region_opened runs in spawned task on open path
Address review feedback: the module/trait docs claimed on_region_opened
runs inline in the region worker loop. That holds for the create path
but not for the open path, where it fires inside the spawned open task
(common_runtime::spawn_global) after WAL replay and before registration.
Correct both the Notes block and the trait method doc so hook authors
don't assume worker-loop-thread affinity or strict ordering on open.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(mito2): fix region_hook inventory and lifecycle wording
Address shuiyisong's review feedback:
- Overview no longer hardcodes 'two methods'; the lifecycle bullet list
now includes on_region_opened.
- on_region_opened is described as firing after open/create succeeds but
before registration (it runs before insert_region on both paths).
- on_region_closed drops the inaccurate 'follower/catchup regions'
exclusion: remove_region fires it for any role, which is consistent
with on_region_opened firing for followers too.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: compare all LoggingOptions fields in PartialEq
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* test: set otlp_export_protocol in load_config_test expected configs
The example configs set otlp_export_protocol = "http", but the expected
LoggingOptions blocks for datanode/frontend/metasrv/standalone left it at
the default None. Now that PartialEq compares all fields, add the field to
match the parsed configs (as the flownode case already does).
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
---------
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report.
Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations.
Signed-off-by: discord9 <discord9@163.com>
- Performance: add collect_lightweight_query_load_metrics to walk the
physical plan and read raw metric values without invoking MetricCollector's
plan-node formatting on the normal query hot path before EOF.
- Refactor: extract collect_full_metrics and keep full aggregation/formatting
for verbose analyze output and terminal metrics.
- Test: cover lightweight partial metrics and drop-time query stats reuse.
Files: src/common/recordbatch/src/adapter.rs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Ensure read requests from `RegionServer` are dispatched to the query
runtime via `common_runtime::spawn_query` instead of executing on the
calling thread. This prevents runtime contention and keeps read
processing on the designated query runtime.
Affected file:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add undrop and purge table procedures
Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen soft-dropped regions before purge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable soft-drop operations for metric logical tables
Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.
Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sync failure detectors during soft-drop lifecycle
Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.
Files:
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/drop_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify soft-drop table tests
Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/key.rs`
- `src/mito2/src/engine/open_test.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: reopen follower regions during undrop
Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: use dropped table ddl expr protos
Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: use inline dropped table task protos
Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.
Files:
- `Cargo.toml`
- `Cargo.lock`
- `src/api/src/helper.rs`
- `src/common/meta/src/rpc/ddl.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: various soft-drop improvements
- Bump `greptime-proto` dependency revision
(`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
(`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
(`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
(`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
(`src/common/meta/src/rpc/ddl.rs`)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: validate table name match in undrop procedure
Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch. This prevents undropping a table by a different name when
only the table ID is known.
- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: simplify UndropTableTask to use table_id only
Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.
Simplify locking in `UndropTableProcedure` to only use `TableLock`.
Update `greptime-proto` dependency revision.
- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: detect table name conflict during tombstone restore in undrop
- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore
- Added `TombstoneTargetAlreadyExists` error variant
- Map tombstone conflict to `TableAlreadyExists` in undrop procedure
- Added test for undrop failing when live name created after prepare
Files:
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/key/tombstone.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: make PurgeDroppedTableTask table_id-only
Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.
- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor
Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.
Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
`TxnOpLimitKvBackend` helper, two new tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge
- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
`open_regions_inner` with an `ignore_region_not_found` flag; expose
`open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
`open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
idempotency and purge replay tolerance of dropped regions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: fix clippy
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(soft-drop): open regions before restoring undrop metadata
Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(tombstone): fail fast on invalid txn op budget
Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump greptime-proto to main branch commit
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
jeremyhi
discord9
QuakeWang
Weny Xu
Lei, HUANG
Yingwen
Ning Sun
shuiyisong
dennis zhuang
raphaelroshan
LFC
fys
sun
sribatsha dash