* fix(metric-engine): handle Utf8View tag/label columns without panicking
label_replace (planned as DataFusion regexp_replace) coerces to Utf8View,
so label columns materialize as StringViewArray; build_tag_arrays'
StringArray downcast then panicked ('tag column must be utf8') — e.g. for
OTLP/json2 ingest. TSID computation, sparse-PK encoding and tag
extraction now accept generic ArrayRef tag columns (Utf8/LargeUtf8/
Utf8View/Dictionary) via string_array_value_at_index, and build_tag_arrays
errors instead of panicking on non-string columns. The mito2 time-series
memtable string-field paths are hardened the same way.
Adds label_replace_with_utf8view_labels_does_not_panic (issue #8732).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(metric-engine): add is_string_null_at helper for tag null checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(datatypes): use is_none_or to satisfy clippy unnecessary-map-or
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix: reject oversized string batches before memtable append
Distinguish a full active string builder from a batch that cannot fit an
empty Arrow string builder at all. Scan every string field so a later
intrinsically oversized field cannot be skipped after an earlier field
requests a freeze. Return InvalidBatch instead of reaching Arrow's offset
overflow panic.
Also cover Utf8View tags with nulls through the metric-engine tag, TSID,
and sparse-primary-key path.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 31a4061552eaa3456faef12b290ed1e10e722804)
(cherry picked from commit 5cef6fe4e14c2847b816009f5838e8e41c9dea94)
When a bulk insert request carries a stale schema, the worker fills the
missing columns via BulkPart::fill_missing_columns before writing. The
method replaced the batch but kept raw_data (the original Arrow Flight
IPC bytes), while BulkWalEntry::try_from(&BulkPart) prefers raw_data, so
the memtable received the filled batch but the WAL recorded the pre-fill
bytes.
Replaying such an entry restores a batch that misses the filled columns:
- Bulk memtable (flat format): convert_bulk_part fails with
ColumnNotFound; the error is swallowed by the no-op write notifier and
the rows are silently lost after restart.
- Time series memtable: BulkPart::to_mutation builds rows shorter than
the declared schema and the region worker panics with index out of
bounds during replay, hanging the region open.
Fixes:
- fill_missing_columns clears raw_data so the WAL entry is re-encoded
from the filled batch.
- replay_memtable fills missing columns for replayed bulk parts of dense
regions, so entries already written by affected versions replay
correctly.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit 1f4a8ad71cdfc135cf571ea7bca2cb87ed90ad72)
(cherry picked from commit 7fae38e667a17b3cd3de3f0540cb3a7b6d5a277a)
* fix(mito2): fail open when Bloom IN predicate has non-literal or unencodable members
collect_in_list previously filtered out non-literal and encoding-failed IN
members and could build a partial hard-pruning predicate, so scanning with
the Bloom filter enabled could prune rows that actually match the query
(false-negative results).
Now any non-literal member or any encoding failure disables Bloom pruning
for the whole IN expression (fail open), while independent AND
subpredicates such as `col = 42` are still extracted as before. Encoding
errors are logged and ignored, so queries never fail.
Adds builder unit tests covering pure-literal, pure-nonliteral, mixed
literal+null, all-null, mixed-nonliteral-with-AND, and encoding-failure
cases, plus an engine-level witness test asserting Bloom-on and Bloom-off
scans return identical rows for a mixed IN filter.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): rename bloom filter tests to semantic names without bug-id prefixes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 70ddf1685eb63d6faca2d0009fe157369c950f73)
(cherry picked from commit 5cbab008ffcdd83c0c5603b81448e1a9b2790254)
* feat: support discarding unflushed region data
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): wake stalled writers after discard
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): drop redundant manifest check for discarding unflushed data
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 62d2c43b54c6bc45de5d49305073e396d6daf434)
* chore(deps): bump datafusion to 452cb4b (support Dictionary literals in substrait)
Backport of #8839 to release/v1.2.
Bump the GreptimeTeam/datafusion fork rev from 6d6ae9a to 452cb4b,
which includes fix(substrait): support Dictionary literals in producer.
This fixes flow queries against dictionary-encoded PK string columns
(metric tables) failing with 'NotImplemented("Unsupported literal:
Dictionary(UInt32, Utf8(...))")'.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito): make do_region_gc pub(crate) to fix private_interfaces warning
clippy runs with -D warnings and this private_interfaces warning fails the
backport CI on release/v1.2.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): fence async index builds by schema generation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retry stale index builds after schema changes
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): split compaction module into scheduler/reader submodules
Extract the compaction scheduler lifecycle (scheduler, status, phases,
execution, SST reservations, pending requests) and its tests out of
compaction.rs into compaction/scheduler.rs and compaction/scheduler_test.rs.
Split the remaining helpers by responsibility:
- estimate_compaction_bytes/refresh_picker_output move to scheduler.rs,
the only call site
- get_expired_ssts moves to picker.rs, shared by the TWCS and window
pickers
- CompactionSstReaderBuilder/time_range_to_predicate/ts_to_lit move to
the new compaction/reader.rs
The root compaction.rs keeps the shared output types and
find_dynamic_options, and re-exports the moved types so existing call
paths stay unchanged. Pure code motion, no behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): separate compaction scheduling from execution details
Turn compaction/scheduler.rs into a directory module to make the
scheduling flow easier to review:
- scheduler.rs keeps the pure scheduling core: the CompactionScheduler
state machine, scheduling entry points, termination chaining, DDL
coordination and region lifecycle events
- scheduler/planning.rs holds the execution-facing parts: background
planning dispatch, picker invocation, plan acceptance, remote/local
submission and memory estimation
- scheduler/state.rs holds the per-region lifecycle types:
CompactionStatus, ActiveCompaction, CompactionPhase, CompactingFiles,
LocalCompactionState, CompactionExecution and PendingCompaction
Child modules keep access to the scheduler's private methods, so the
split is pure code motion with minimal visibility changes (pub(super)
only where the parent module or tests reach into child items).
No behavior change.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): use absolute scheduler imports
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): document compaction scheduler modules
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): make async index publication conditional
Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale.
Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(mito2): centralize manifest update finalization
Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): retain index build leases across reopen
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid retiring scheduler on index failure
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): handle cross-region index publication
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): use physical region for index paths
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(cmd): update noop index builder
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): avoid reusing published index versions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore(mito2): log untracked index build stops
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(mito2): set compaction time range in index test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: expose MitoRegion::all_manifest_files for metadata rebuild
Adds a public read-only accessor that returns all live SST file metas
and the current manifest version from the region manifest. Used by the
downstream project admin path to enumerate the authoritative live file
set without going through the worker loop.
* fix(mito2): merge staging manifest files in all_manifest_files
The original implementation only read the normal manifest
(manifest_ctx.manifest()) and skipped staging_manifest(). While the
region is in staging mode (e.g. region copy/migration), the authoritative
live file set lives in the staging manifest, so callers would silently
miss those files.
Now matches the semantics of manifest_sst_entries() (~L771), which
explicitly merges manifest().files with staging_manifest().files via a
HashMap collect (dedup by FileId). The returned manifest version is the
staging version when a staging manifest is present, otherwise the normal
manifest version.
Also removed downstream-specific references from the rustdoc comments.
* refactor(mito2): run compaction picking in background with plan tracking
Move the compaction picker out of the region worker's critical path by
dispatching planning to a background task and reporting the result back
via CompactionPickFinished. CompactionStatus now tracks an explicit
picking phase keyed by a monotonic plan id, so stale planning results
are rejected and duplicate regular triggers coalesce while picking.
Before submitting a prepared compaction, the picker output is refreshed
against the current SST version (file handles are re-resolved and
conflicts roll back reservations), ensuring the plan still matches live
state. CompactionExecution identifies the running task by
(plan id, kind, version control) so finish/cancel/fail notifications
from outdated executions are ignored.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): notify pick finished even when compaction planning panics
The worker only leaves the picking phase after receiving the
CompactionPickFinished notification. Previously the planning task was
spawned fire-and-forget: if it panicked before sending the notification,
the region would be stuck in the picking phase forever, blocking all
future compactions and pending DDLs (e.g. entering staging) of the
region.
Wrap the planning future with catch_unwind so a panic is converted into
a CompactionPlanningResult::Error and the notification is always sent,
letting the worker run the normal error cleanup path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): dec inflight compaction gauge after re-entrancy guard
DefaultNotifier::notify decremented INFLIGHT_COMPACTION_COUNT before the
re-entrancy guard, so a duplicate notify (which should never happen, but
the guard exists to defend against it) would decrement the gauge an
extra time and let it drift negative. Move the decrement after the
guard, matching the local compaction path's guard-then-account order.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): remove idle compaction status to prevent zombie state
When a compaction finished within min_compaction_interval with no
pending requests, on_compaction_finished left an idle status
(phase = None) behind. The worker then skipped chaining the next
compaction due to the interval gate, and the leftover status made
schedule_compaction swallow all future triggers of the region: regular
waiters were queued but never woken, and manual requests stayed pending
forever. The region stopped compacting until close/drop/truncate.
Add CompactionScheduler::remove_idle_status and call it from
handle_compaction_finished when the interval has not elapsed and no
chained planning is scheduled. The chain-until-no-plan semantics for
compactions that outlast the interval is preserved.
Also drops an unused import left by the previous commit.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): make compaction scheduling methods synchronous
schedule_compaction, handle_pending_compaction_request and
schedule_next_compaction no longer await anything after compaction
planning became fire-and-forget. Drop the async signature to make the
no-suspension-point invariant explicit: these methods always run to
completion on the worker loop without reentrancy.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): report stale compaction execution instead of region closed
When a compaction finishes but its execution no longer matches the
current one, the region may have been reopened or truncated, or the
compaction was superseded. Reporting RegionClosed to waiters is
misleading; introduce a neutral StaleCompactionExecution error (same
Cancelled status code) for this case.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): serialize truncate with compaction
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): avoid panic-based compaction status lookups
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): keep in-flight compaction plan when scheduling next
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): notify cancelled compaction pending ddl
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* typo: rename prefence to pre_fence to bypass typo check
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): trim redundant compaction tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): move compaction tests to dedicated file
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo and format
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "test(mito2): move compaction tests to dedicated file"
This reverts commit e202f1f5
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: revert test movement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): remove redundant compaction status lookups
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): remove duplicate compaction test file
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): prune redundant compaction tests
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): simplify compaction plan identity
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): design pending regular state simplification
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): plan pending regular state simplification
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): simplify pending regular compaction state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* remove: plan files
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify compaction completion handling
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): inline compaction phase execution lookup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): warn instead of panic on pending DDL for non-compacting region
add_ddl_request_to_pending unwrapped the region status and panicked when
the region was not compacting. Log a warning and skip the request instead,
and inline the now-trivial CompactionStatus::queue_ddl helper.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): dispatch pending DDLs before chaining regular compaction
A DDL queued behind a TooLateToCancel compaction (commit started or
remote execution) was deferred behind a whole extra plan/execution
cycle when a regular trigger had been retained during picking. Dispatch
the pending DDLs as soon as the current task finishes instead: satisfy
the retained regular waiters with the just-finished compaction, remove
the region status, and return the DDLs immediately.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): demote stale compaction failure logs to debug
Check region presence and execution staleness before logging, so a
superseded execution's terminal failure no longer emits a misleading
error log.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): atomically queue compaction DDLs
Combine compaction cancellation and dependent DDL enqueueing under one
status borrow. Return the typed request unchanged when no compaction is
running, avoiding both an unreachable warning branch and silent DDL
loss if the invariant changes.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): explain why picker output handles are re-resolved
Addresses review question on refresh_picker_output: picking runs in
background on a possibly-stale version snapshot, so handles must be
re-resolved against the current version at accept time to detect
removed files and to read/reserve the up-to-date handle.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): keep compaction gate in test module
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(mito2): format compaction DDL helper calls
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): group active compaction state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): fence compaction triggers behind pending DDL
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): simplify pending DDL collection in compaction scheduler
Replace the take-and-restore dance of the active compaction state with
an up-front busy check before handling pending compaction requests,
then take the active state once to drain DDL waiters.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: clarify that pending_request only carries manual StrictWindow compaction in production
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): arm DDL gate before cancellation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
When a compaction finishes with no pending request/DDL, on_compaction_finished
leaves the region's CompactionStatus in the map with no active compaction. If
the worker then skips schedule_next_compaction because min_compaction_interval
has not passed, the stale status is never cleaned up: all subsequent compaction
requests are swallowed by the "status exists" branch (regular waiters hang,
manual StrictWindow requests pend forever) and the region never compacts again
until restart.
Add CompactionScheduler::remove_inactive_status() and call it from
handle_compaction_finished when the interval gate skips scheduling the next
compaction, restoring the invariant that a status in the map always has an
active compaction.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: add field id and extension type to histogram
* chore: revert histogram check
* chore: lint
* test: add test coverage for maybe_update_schema
* feat: calculate sub field id from parent column id
* refactor(mito2): guard histogram sub-field ids and cover parquet footer
Address review on #8528:
- native_histogram: derive sub-field ids with checked arithmetic. The SST
writer now surfaces a new InvalidNativeHistogramSubfield error when a
sub-field id cannot be resolved (unknown name or i32 overflow) instead of
silently dropping the id or wrapping. maybe_wrap_schema is now fallible.
- sst: add parquet writer/footer round-trip tests asserting the
greptime.histogram extension and nested PARQUET:field_id (incl. list
elements) survive on disk, and a non-canonical struct is left untouched.
- sst: fix a broken rustdoc link to stamp_native_histogram_subfield_ids.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return error when fail to get i32 column id
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): skip reopening dropped tables during purge
Purge soft-dropped tables by dropping stored routes directly instead of
reopening tombstoned regions first. Treat legacy
\`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only
transition to \`DropRegions\`.
Files:
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): guard undrop restore race
Serialize `UNDROP TABLE` with same-name creates by seeding the
original table name before procedure submission, and clean up reopened
regions when metadata restore fails.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(datanode): reject cleanup for registered regions
Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping.
Add coverage for `OfflineCleanup` engine selection and registered-region rejection.
Files:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): require tombstone before undrop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: harden soft-drop cleanup paths
Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop.
Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup.
Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/mito2/src/engine/open_test.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump cleanup proto dependency
Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload.
Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/store-api/src/region_request.rs\`
- \`src/common/meta/src/ddl/drop_table/executor.rs\`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: share region cleanup helpers
Share common region cleanup helpers between normal drop and offline cleanup
while keeping their preconditions separate.
- Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`.
- Share runtime artifact and manifest cache cleanup after full deletion paths.
- Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC.
Files:
- `src/mito2/src/worker/handle_drop.rs`
- `src/mito2/src/worker/handle_open.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: flush soft-dropped regions on close
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): handle flush-on-close race with concurrent in-flight flush
When a region close with `flush_on_close: true` races with an
already-running flush, pass the actual close request (including the
flush_on_close flag) to the DDL handler instead of a default request
so the pending flush is correctly awaited.
Files: `src/mito2/src/worker/handle_close.rs`
Also adds a test verifying that closing with flush-on-close while a
flush is in progress still persists all written data correctly.
Files: `src/mito2/src/engine/close_test.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: finish close request callsites
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: replace imperative close-region loop with functional combinators
Transform the region close dispatch in `DropTableExecutor` from mutable
`Vec` and `push` loops to iterator chains with `join_all`, improving
idiomatic Rust style and readability.
- `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode
region-close logic to use `peers.map()` and nested `join_all`, moving
`node_manager.datanode()` inside the closure to align with the new
structure
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: decouple Kafka client from WAL checkpoint
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: merge Kafka WAL index checkpoints
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: delegate Kafka WAL retirement to metasrv
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: rebase main and resolve conflicts
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: license header
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: bump proto to commits on main
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove Kafka obsolete-all index changes
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Only fire `on_manifest_updated` for writes to the live (normal) manifest
directory. Writes to the staging directory (enter staging, operations during
staging, the intermediate apply-staging `RegionEdit`) are suppressed — their
effects accumulate and are delivered in a single notification when
`exit_staging_on_success` promotes all staged actions to the live manifest.
This simplifies hook implementations that only care about "live" state, since
they no longer receive intermediate staging notifications they cannot act on,
and no file-list information is lost (the promote step carries all accumulated
files from staging operations).
Key changes:
- `PendingManifestHook` gains an `is_staging` field; `fire()` is a no-op when
`is_staging` is `true`
- `ManifestContext::update_locked` threads its `is_staging` parameter through
- Updated module-level docs and coverage table
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: hook extension for region/file close/drop
* feat: cover region lifecycle with open/close/drop hooks
* docs(mito2): clarify on_region_opened runs in spawned task on open path
Address review feedback: the module/trait docs claimed on_region_opened
runs inline in the region worker loop. That holds for the create path
but not for the open path, where it fires inside the spawned open task
(common_runtime::spawn_global) after WAL replay and before registration.
Correct both the Notes block and the trait method doc so hook authors
don't assume worker-loop-thread affinity or strict ordering on open.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(mito2): fix region_hook inventory and lifecycle wording
Address shuiyisong's review feedback:
- Overview no longer hardcodes 'two methods'; the lifecycle bullet list
now includes on_region_opened.
- on_region_opened is described as firing after open/create succeeds but
before registration (it runs before insert_region on both paths).
- on_region_closed drops the inaccurate 'follower/catchup regions'
exclusion: remove_region fires it for any role, which is consistent
with on_region_opened firing for followers too.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Skip scheduled meta GC while cluster maintenance mode is enabled and reject manual GC requests explicitly instead of returning an empty success report.
Also increase mito GC's default lingering time to 1h and update generated config docs and config API expectations.
Signed-off-by: discord9 <discord9@163.com>
* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
discord9
jeremyhi
fys
LFC
Yingwen
Lei, HUANG
Ning Sun
dennis zhuang
shuiyisong
Weny Xu
Ruihang Xia
sribatsha dash