* fix(meta): skip reopening dropped tables during purge
Purge soft-dropped tables by dropping stored routes directly instead of
reopening tombstoned regions first. Treat legacy
\`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only
transition to \`DropRegions\`.
Files:
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): guard undrop restore race
Serialize `UNDROP TABLE` with same-name creates by seeding the
original table name before procedure submission, and clean up reopened
regions when metadata restore fails.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(datanode): reject cleanup for registered regions
Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping.
Add coverage for `OfflineCleanup` engine selection and registered-region rejection.
Files:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): require tombstone before undrop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: harden soft-drop cleanup paths
Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop.
Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/mito2/src/engine/open_test.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump cleanup proto dependency
Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/store-api/src/region_request.rs\`
- \`src/common/meta/src/ddl/drop_table/executor.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: share region cleanup helpers
Share common region cleanup helpers between normal drop and offline cleanup
while keeping their preconditions separate.
- Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`.
- Share runtime artifact and manifest cache cleanup after full deletion paths.
- Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC.
Files:
- `src/mito2/src/worker/handle_drop.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(flow): convert streaming expiration to milliseconds
Flow EXPIRE AFTER values are stored in seconds, while streaming flow timestamps and durations use milliseconds. Passing the value through unchanged expires state and limits refill scans 1000 times too early.
Convert the value at streaming create and refill boundaries, reject overflow, and document the CreateFlowArgs unit.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* fix(flow): reject negative streaming expiration
Reject negative EXPIRE AFTER values at the streaming adapter boundary so refill ranges cannot move into the future. Keep zero valid and retain checked seconds-to-milliseconds conversion.
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
---------
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* fix: flush soft-dropped regions on close
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): handle flush-on-close race with concurrent in-flight flush
When a region close with `flush_on_close: true` races with an
already-running flush, pass the actual close request (including the
flush_on_close flag) to the DDL handler instead of a default request
so the pending flush is correctly awaited.
Files: `src/mito2/src/worker/handle_close.rs`
Also adds a test verifying that closing with flush-on-close while a
flush is in progress still persists all written data correctly.
Files: `src/mito2/src/engine/close_test.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: finish close request callsites
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: replace imperative close-region loop with functional combinators
Transform the region close dispatch in `DropTableExecutor` from mutable
`Vec` and `push` loops to iterator chains with `join_all`, improving
idiomatic Rust style and readability.
- `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode
region-close logic to use `peers.map()` and nested `join_all`, moving
`node_manager.datanode()` inside the closure to align with the new
structure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: decouple Kafka client from WAL checkpoint
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: merge Kafka WAL index checkpoints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: delegate Kafka WAL retirement to metasrv
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: rebase main and resolve conflicts
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: license header
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove Kafka obsolete-all index changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Only fire `on_manifest_updated` for writes to the live (normal) manifest
directory. Writes to the staging directory (enter staging, operations during
staging, the intermediate apply-staging `RegionEdit`) are suppressed — their
effects accumulate and are delivered in a single notification when
`exit_staging_on_success` promotes all staged actions to the live manifest.
This simplifies hook implementations that only care about "live" state, since
they no longer receive intermediate staging notifications they cannot act on,
and no file-list information is lost (the promote step carries all accumulated
files from staging operations).
Key changes:
- `PendingManifestHook` gains an `is_staging` field; `fire()` is a no-op when
`is_staging` is `true`
- `ManifestContext::update_locked` threads its `is_staging` parameter through
- Updated module-level docs and coverage table
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: hook extension for region/file close/drop
* feat: cover region lifecycle with open/close/drop hooks
* docs(mito2): clarify on_region_opened runs in spawned task on open path
Address review feedback: the module/trait docs claimed on_region_opened
runs inline in the region worker loop. That holds for the create path
but not for the open path, where it fires inside the spawned open task
(common_runtime::spawn_global) after WAL replay and before registration.
Correct both the Notes block and the trait method doc so hook authors
don't assume worker-loop-thread affinity or strict ordering on open.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(mito2): fix region_hook inventory and lifecycle wording
Address shuiyisong's review feedback:
- Overview no longer hardcodes 'two methods'; the lifecycle bullet list
now includes on_region_opened.
- on_region_opened is described as firing after open/create succeeds but
before registration (it runs before insert_region on both paths).
- on_region_closed drops the inaccurate 'follower/catchup regions'
exclusion: remove_region fires it for any role, which is consistent
with on_region_opened firing for followers too.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: compare all LoggingOptions fields in PartialEq
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
* test: set otlp_export_protocol in load_config_test expected configs
The example configs set otlp_export_protocol = "http", but the expected
LoggingOptions blocks for datanode/frontend/metasrv/standalone left it at
the default None. Now that PartialEq compares all fields, add the field to
match the parsed configs (as the flownode case already does).
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
---------
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report.
Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations.
Signed-off-by: discord9 <discord9@163.com>
- Performance: add collect_lightweight_query_load_metrics to walk the
physical plan and read raw metric values without invoking MetricCollector's
plan-node formatting on the normal query hot path before EOF.
- Refactor: extract collect_full_metrics and keep full aggregation/formatting
for verbose analyze output and terminal metrics.
- Test: cover lightweight partial metrics and drop-time query stats reuse.
Files: src/common/recordbatch/src/adapter.rs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Ensure read requests from `RegionServer` are dispatched to the query
runtime via `common_runtime::spawn_query` instead of executing on the
calling thread. This prevents runtime contention and keeps read
processing on the designated query runtime.
Affected file:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
shuiyisong
discord9
Weny Xu
jeremyhi
Ruihang Xia
Lei, HUANG
LFC
QuakeWang
Yingwen
Ning Sun
dennis zhuang
raphaelroshan
fys
sun
sribatsha dash