mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-14 16:03:27 +00:00
fa128adb8ed2a7948bc2235cb9b1fe2a6fbfec76
424
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Lei, HUANG
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 discord9
|
7cf84892d2 |
perf(table): filter decoded rows with dynamic predicates (#9004)
* perf(table): filter decoded rows with dynamic predicates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve unknown dynamic filter rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(table): use null guards for dynamic filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve null inputs during dynamic pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover frontend join dynamic filter transfer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): reset dynamic filters and scanner pruning state Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
 Yingwen
|
28e2b5aca2 |
feat: expose missing SST manifest fields (#8965)
* feat: expose missing SST manifest fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve SST manifest column ordinals Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
78084a9d44 |
feat: add admin function to discard unflushed data (#8768)
* feat: add admin function to discard unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover discarding unflushed data by table Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix license header Signed-off-by: evenyag <realevenyag@gmail.com> * fix: reject discarding logical metric table data Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: defer table name formatting in error paths Signed-off-by: evenyag <realevenyag@gmail.com> * chore(deps): update greptime-proto revision Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename discard unflushed admin function Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 jeremyhi
|
c7c1a61d56 |
feat: support enabling skip_wal with ALTER TABLE (#8730)
* feat: support enabling skip_wal with alter table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover skip wal on metric physical region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: retry skip wal alter on route changes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address skip wal review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve skip wal create options Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: prefer typed skip wal option Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address skip wal review comments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: canonicalize legacy skip wal option Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep typed skip wal canonical Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: simplify skip wal option tracking Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
c59ec0154e |
feat(mito2): discard unflushed region data safely (#8600)
* feat: support discarding unflushed region data Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): wake stalled writers after discard Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): drop redundant manifest check for discarding unflushed data Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
f524a0b5b4 |
feat: support time range in manual compaction (#8669)
* feat: support time range in manual compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: reject overflowing compaction range alignment Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: preserve range across compaction continuations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf: use graph traversal for compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commit on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain compaction window dependency closure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
3e67c67607 |
refactor: simplify scan projection to root column indices (#8629)
* refactor: simplify scan projection to root column indices Signed-off-by: fys <fengys1996@gmail.com> * fix: sqlness test Signed-off-by: fys <fengys1996@gmail.com> * chore: code style adjust Signed-off-by: fys <fengys1996@gmail.com> * fix: cr Signed-off-by: fys <fengys1996@gmail.com> * add comment and test Signed-off-by: fys <fengys1996@gmail.com> * minor change Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Yingwen
|
9b579c7618 |
feat: make parquet row group size configurable (#8446)
* feat: make parquet row group size configurable via max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * feat: split inverted index segments across row group boundaries Signed-off-by: evenyag <realevenyag@gmail.com> * test: verify max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * test: update row group size sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): use parquet row group size for index rebuild Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update query perf fixture indexer builder Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clamp index selection to parquet rows Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support altering parquet row group size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle idempotent append mode in mixed alters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): stage mixed region option alters atomically Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): ignore inverted indexes with mismatched row counts Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Ruihang Xia
|
0479c05092 |
perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(client): skip dictionary Flight batches Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): decode dictionary labels in HTTP output Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test(mito2): support dictionary tags in series scans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(servers): preserve dictionary child nulls in SQL Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(promql): compare dictionary tags by logical nulls Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): keep dictionary tags within query paths Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor(query): scope PK dictionary encoding to reads Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve pushdown for dictionary labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): handle dictionary labels in query operators Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): complete label type matching Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): align dictionary query schemas Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): decode incompatible OR labels Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix(query): preserve dictionary partition pruning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: handle dictionary query edge cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: import dictionary downcast macro Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Yingwen
|
0a002f4d65 |
feat: support per-region write buffer limits (#8473)
* feat(mito): add per-region write buffer limit Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): add default region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * docs: expand configuration change checklist Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover table write buffer size option Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): refine region write buffer checks Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): keep region-stalled writes queued Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): allow zero region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate region state before write stall Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): reject writes beyond region hard limit Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update example Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
81d8beee3a |
feat: clean up soft-dropped regions offline (#8458)
* fix(meta): skip reopening dropped tables during purge Purge soft-dropped tables by dropping stored routes directly instead of reopening tombstoned regions first. Treat legacy \`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only transition to \`DropRegions\`. Files: - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): guard undrop restore race Serialize `UNDROP TABLE` with same-name creates by seeding the original table name before procedure submission, and clean up reopened regions when metadata restore fails. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl_manager.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(datanode): reject cleanup for registered regions Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping. Add coverage for `OfflineCleanup` engine selection and registered-region rejection. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): require tombstone before undrop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: harden soft-drop cleanup paths Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop. Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/mito2/src/engine/open_test.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump cleanup proto dependency Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/store-api/src/region_request.rs\` - \`src/common/meta/src/ddl/drop_table/executor.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share region cleanup helpers Share common region cleanup helpers between normal drop and offline cleanup while keeping their preconditions separate. - Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`. - Share runtime artifact and manifest cache cleanup after full deletion paths. - Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC. Files: - `src/mito2/src/worker/handle_drop.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
56e9158819 |
feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): handle flush-on-close race with concurrent in-flight flush When a region close with `flush_on_close: true` races with an already-running flush, pass the actual close request (including the flush_on_close flag) to the DDL handler instead of a default request so the pending flush is correctly awaited. Files: `src/mito2/src/worker/handle_close.rs` Also adds a test verifying that closing with flush-on-close while a flush is in progress still persists all written data correctly. Files: `src/mito2/src/engine/close_test.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: finish close request callsites Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: replace imperative close-region loop with functional combinators Transform the region close dispatch in `DropTableExecutor` from mutable `Vec` and `push` loops to iterator chains with `join_all`, improving idiomatic Rust style and readability. - `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode region-close logic to use `peers.map()` and nested `join_all`, moving `node_manager.datanode()` inside the closure to align with the new structure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: decouple Kafka client from WAL checkpoint Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: merge Kafka WAL index checkpoints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: delegate Kafka WAL retirement to metasrv Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: rebase main and resolve conflicts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: license header Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commits on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove Kafka obsolete-all index changes Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
03b304522f |
refactor: model distributed inspect fan-in as exec (#8447)
* refactor: model distributed inspect fan-in as exec Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden distributed inspect exec fan-in Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: show distributed inspect scan in plans Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stabilize distributed inspect sqlness Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: apply sqlness replacement to inspect explain Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: redact inspect repartition plan Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 sribatsha dash
|
8f5d681e4b |
feat: support ALTER TABLE SET auto_flush_interval (#8403)
* feat: support ALTER TABLE SET auto_flush_interval Closes #8394. Add a new SetRegionOption::AutoFlushInterval variant so that the per-table auto flush interval can be changed on existing tables via 'ALTER TABLE t SET ...', following up the CREATE TABLE path from #8357. - region_request.rs: parse 'auto_flush_interval' with humantime and map it to the new variant. - metadata.rs: persist the value (or remove it, if None) in TableOptions.extra_options using the same humantime string format the engine already expects. - handle_alter.rs: apply the new interval in handle_alter_region_options_fast (no memtable flush needed, same pattern as Ttl) and group the variant with Ttl/Twsc in new_region_options_on_empty_memtable. Tests: - Two unit tests in metadata.rs covering set and unset-to-None. - A new sqlness case alter_auto_flush_interval.sql covering create-then-alter, alter-then-alter, invalid duration, and alter on a table that already had auto_flush_interval at create time. Signed-off-by: srivtx <crypticcc101@gmail.com> * fix: validate auto_flush_interval > 0 in ALTER SET path Gemini code assist flagged that the request parser accepted a zero duration, leaving the rejection to the downstream RegionOptions validation which only fires on next flush. Reject it at parse time so users get the error immediately at the ALTER TABLE statement. Also add a '0s' error case to the sqlness test. Signed-off-by: srivtx <crypticcc101@gmail.com> * fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result Address the rest of fengjiachun's review on #8403: 1. Empty value in ALTER SET clears the override (parallels Ttl). 'ALTER TABLE t SET ... = NULL' comes through as value = ''; we now return AutoFlushInterval(None) so the override is removed from TableOptions.extra_options, matching the Ttl pattern. 2. Add a unit test in region_request.rs covering the four cases (valid, empty-clears, zero-rejected, garbage-rejected). 3. Generate and check in alter_auto_flush_interval.result via 'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both the standalone and distributed sqlness jobs now pass locally, and the test extension covers the NULL-clears path end to end. Signed-off-by: srivtx <crypticcc101@gmail.com> --------- Signed-off-by: srivtx <crypticcc101@gmail.com> |
||
|
Weny Xu
|
2dc6addb66 |
feat: report region query stats in heartbeat (#8401)
* feat: report region read load in heartbeat Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: expose region query stats in information schema Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update sqlness result Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: record region query stats on stream drop Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: keep region query cpu stats in nanoseconds Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 raphaelroshan
|
4c28afd16f |
feat: support table-level auto_flush_interval (#8357)
Make auto_flush_interval configurable per table (region) in addition to
the global Mito engine config, mirroring how ttl supports a global default
with a per-table override.
- Add auto_flush_interval to RegionOptions, parsed from the table option
with the same humantime format as the global config; reject a
non-positive value.
- The periodic flush logic resolves the effective interval per region,
falling back to the global config when unset, using a saturating
conversion to avoid overflow on extreme values.
- Accept the option key in is_mito_engine_option_key so it can be set at
CREATE TABLE via WITH ('auto_flush_interval' = '5m').
Ref #8340
Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
|
||
 Lei, HUANGandLei, HUANG
|
9b1672316c |
fix(metric-engine): report query load under physical region id (#8355)
* fix(metric-engine): report query load under physical region id Propagate the physical region ID through the scanner, record batch stream, and query engine so that query-load metrics (CPU time, scanned bytes) are attributed to the correct physical region rather than always to the logical region. - `src/store-api/src/region_engine.rs` — add `query_load_region_id` to `ScannerProperties` and `set_query_load_region_id` to `RegionScanner` trait - `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` — implement the new trait method on each scanner - `src/common/recordbatch/src/adapter.rs` — carry region id through `RecordBatchStreamAdapter` into `RecordBatchMetrics` - `src/table/src/table/scan.rs` — expose region id on `RegionScanExec` - `src/query/src/datafusion.rs` — extract region id from the physical plan and set it on the output stream - `src/query/src/dist_plan/merge_scan.rs` — use metrics-contained region id in query-load reporting, falling back to the logical region id - `src/metric-engine/src/engine/read.rs` — set region id on the metric engine scanner Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(query): satisfy clippy for query load region id Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(query): ignore missing query load region ids Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
758a166325 |
fix: include index files in GC listing (#8327)
* fix: include index files in GC listing Signed-off-by: discord9 <discord9@163.com> * chore: filter GC index listing to puffins Signed-off-by: discord9 <discord9@163.com> * chore: simplify GC index listing stream Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
Weny Xu
|
1ca29c3481 |
feat: support create region requirements (#8281)
* feat: support create region requirements Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: require object storage for repartition creates Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: upgrade proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 dennis zhuang
|
ca21c9f048 |
fix!: correct information_schema index metadata (#8275)
fix: correct information_schema index metadata Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
da9d314fa4 |
fix: align bulk insert schema handling (#8222)
* fix: align bulk insert schema handling Carry `aligned_schema_version` through bulk insert requests so datanode can recheck stale schemas only when needed. Use the physical data region schema version for metric logical bulk inserts, while external bulk insert callers leave alignment unknown. Related files: - `Cargo.toml` - `Cargo.lock` - `src/store-api/src/region_request.rs` - `src/metric-engine/src/engine/bulk_insert.rs` - `src/mito2/src/request.rs` - `src/mito2/src/worker/handle_bulk_insert.rs` - `src/mito2/src/worker/handle_write.rs` - `src/operator/src/bulk_insert.rs` - `src/servers/src/pending_rows_batcher.rs` - `src/mito2/src/engine/edit_region_test.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: cover bulk insert missing columns Add Flight bulk insert coverage for batches that omit nullable columns and verify the database fills the missing values with nulls. Related files: - `tests-integration/src/grpc/flight.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: update proto to main branch commit Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: align physical metric bulk insert schema Pass the physical schema version through metric bulk insert requests so physical and logical region writes share the same schema alignment behavior. Files: - \`src/metric-engine/src/engine/bulk_insert.rs\` — updates \`bulk_insert_physical_region\` and reuses \`physical_schema_version\` for bulk insert schema alignment. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: add some doc Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Weny Xu
|
935ef9a361 |
feat: check open region requirements (#8194)
* feat: check open region capabilities Signed-off-by: WenyXu <wenymedia@gmail.com> * test: allow file region migration tests Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refine open region requirements Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use fs scheme constant Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover open region requirement predicate Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: check file engine open requirements Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
1a2d046351 |
fix(mito): count owned SSTs in region stats (#8191)
* fix(mito): count owned SSTs in region stats Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): use origin region for index metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
9a4e5e8457 |
chore: expose region info inspection table (#8178)
* chore/region-sync-diff: add region info inspection core - `store-api`: add `RegionInfoEntry` schema and plan builder in `src/store-api/src/region_info.rs` and export it from `src/store-api/src/lib.rs` - `mito2`: collect region runtime metadata with `MitoEngine::all_region_infos` and `RegionRoleState::as_str` in `src/mito2/src/engine.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, `src/mito2/Cargo.toml`, and `Cargo.lock` - `datanode`: expose the reserved `InspectRegionInfo` provider in `src/datanode/src/region_server/catalog.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: expose region info schema table - `information_schema.region_info`: add frontend table wiring in `src/catalog/src/system_schema/information_schema.rs`, `src/catalog/src/system_schema/information_schema/region_info.rs`, `src/catalog/src/system_schema/information_schema/table_names.rs`, and `src/common/catalog/src/consts.rs` - `region_group` removal: drop `region_group` from `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, and `src/mito2/src/engine/basic_test.rs` - `SQLness coverage`: add standalone coverage in `tests/cases/standalone/common/information_schema/region_info.sql` and `tests/cases/standalone/common/information_schema/region_info.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: restore region group info - `region_info` schema: restore `region_group` alongside `region_sequence` in `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, and `tests/cases/standalone/common/information_schema/region_info.result` - `MitoEngine::all_region_infos`: remove redundant iterator conversion in `src/mito2/src/engine.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: clarify region sequence columns - `region_info` schema: rename `sequence` to `committed_sequence` and add nullable `flushed_sequence` in `src/store-api/src/region_info.rs` and `src/mito2/src/region.rs` - `region_info` coverage: update sequence assertions and expected metadata in `src/mito2/src/engine/basic_test.rs`, `tests/cases/standalone/common/information_schema/region_info.sql`, `tests/cases/standalone/common/information_schema/region_info.result`, and `tests/cases/standalone/common/system/information_schema.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: report region options errors - `region_info` output: preserve `region_options` serialization failures as JSON error objects in `src/mito2/src/region.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
discord9
|
eccd97b5c7 |
feat(flow): support incremental read checkpoints (#8179)
* feat: flownode inc mode Signed-off-by: discord9 <discord9@163.com> * chore: rename fallback reason Signed-off-by: discord9 <discord9@163.com> * fix: harden flow incremental checkpoints Signed-off-by: discord9 <discord9@163.com> * fix: address flow watermark lint Signed-off-by: discord9 <discord9@163.com> * fix: address flow clippy Signed-off-by: discord9 <discord9@163.com> * refactor: clarify incremental plan preparation Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: more sqlness test Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
15fc148e40 |
refactor: clarify region flush reasons (#8146)
* refactor: clarify region flush reasons Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 LFC
|
a6dc55a71c |
feat: show sst primary key range in information_schema (#8137)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
7840aa1bb4 |
refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk Signed-off-by: evenyag <realevenyag@gmail.com> * chore: keep partition tree memtable for migration test Restore PartitionTreeMemtable construction when memtable.type=partition_tree is explicit, and move the sparse-encoding bulk override into the default (no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse, so the test actually exercises a PartitionTreeMemtable in phase 1 and verifies WAL replay into the new BulkMemtable on reopen without flushing. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: drop partition tree memtable from runtime Re-apply the unconditional sparse-encoding override in `MemtableBuilderProvider::builder_for_options` and route the `MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation warning. After this change, `PartitionTreeMemtableBuilder` is no longer reachable from the engine runtime; benchmarks still reference the type. Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the `put_sparse_rows` helper added in the previous commit — that test only existed to validate the PartitionTree -> Bulk reopen migration and is unnecessary now that the override is in place. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): move timestamp_array_to_i64_slice into read module Relocate the timestamp_array_to_i64_slice helper from memtable/partition_tree/data.rs to the read module so that the read path no longer depends on the partition_tree internals. All call sites (both inside and outside the partition_tree module) now import from crate::read. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests The time_partition tests use the memtable builder purely as a generic backend for the TimePartitions write/scan paths; nothing in them is specific to the partition-tree memtable. Switch the seven affected tests to TimeSeriesMemtableBuilder so the tests no longer depend on PartitionTreeMemtableBuilder. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): delete PartitionTreeMemtable implementation The runtime already falls back to BulkMemtable for the PartitionTree variant. Drop the now-unreachable implementation, its metrics, the partition_tree benchmarks, the metric-engine Unsupported fallback in bulk_insert.rs, and the test helpers that only existed for the deleted module. MemtableOptions::PartitionTree, its parsing, the runtime fallback, the store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures remain so existing region options keep round-tripping. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito-codec): drop skip_partition_column parameter PartitionTreeMemtable was the only caller passing skip_partition_column=true; every other caller passes false. Now that the partition_tree module is gone, the parameter is uniformly false and the guard branch is dead. Drop the parameter from the trait method and both impls, remove the guard and the is_partition_column helper, and update the four remaining call sites in mito2 plus the bench. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove unused MemtableConfig enum Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused variant Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test_config_api Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove unused memtable test helpers Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: support bulk memtable options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove partition tree options from region options Move primary_key_encoding to the top level Signed-off-by: evenyag <realevenyag@gmail.com> * test: make ssts test datetime replaced text stable Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * chore: validate_enum_options consider bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass region id when parsing region options Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with `try_from_options(region_id, options_map)` so the legacy partition_tree fallback can log the affected region. The fallback now also overrides the SST format to flat in addition to clearing the memtable type. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: align sst_format with bulk memtable on parse and open Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
923c4227a3 |
feat: remote dyn filter basics (#7979)
* feat: dyn filter update abi Signed-off-by: discord9 <discord9@163.com> * feat: add remote dyn filter region rpc scaffolding Signed-off-by: discord9 <discord9@163.com> * feat: query id Signed-off-by: discord9 <discord9@163.com> * fix: prevent spoof Signed-off-by: discord9 <discord9@163.com> * fix: more proofing Signed-off-by: discord9 <discord9@163.com> * feat: make payload base64 Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * update Signed-off-by: discord9 <discord9@163.com> * chore: resolve conflict Signed-off-by: discord9 <discord9@163.com> * chore: update to main Signed-off-by: discord9 <discord9@163.com> * fix: add back defensive check Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
LFC
|
7a285c2890 |
feat: concretize json type from query (#8081)
* feat: concretize json type from query Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * add more tests Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 QuakeWang
|
45e990b7f3 |
refactor: propagate flush reasons through FlushRegions path (#8051)
* feat: propagate flush reasons through FlushRegions path Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: address flush reason review feedback Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: keep flush instruction helper name Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
fys
|
c90e4147de |
refactor: introduce the ProjectInput structure (#7908)
* refactor: introduce the ProjectInput structure * remove unused import * fix: cr * fix: cr * fix: code review * add more unit test * avoid clone of input.projection |
||
|
Weny Xu
|
57f1921253 |
feat: propagate staging leader through lease and heartbeat (#7950)
* feat(mito): expose staging leader role state * fix(region): clear staging metadata on leader exit * feat: propagate staging leader role through heartbeat and metasrv * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(region): unify staging exit role transitions * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
2af59ed386 |
feat: always use flat scan path for both format (#7901)
* feat: remove primary_key format scan path Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove flat format flag Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove CompatReader tests Signed-off-by: evenyag <realevenyag@gmail.com> * chore: show whether the format is flat in explain Signed-off-by: evenyag <realevenyag@gmail.com> * test: stable series scan result Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Ning Sun
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
 discord9
|
d7bc5ad16b |
feat: add incremental read context and scan boundaries (#7848)
* feat: add incremental read context and scan boundaries Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * docs: explain field Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
jeremyhi
|
8058ce7cf2 |
refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make confg-docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by codex review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: track_with_policy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: mem granularity mb to kb Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by scan_memory_on_exhausted comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: typo Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Ruihang Xia
|
2af3951944 |
feat: cache decoded region metadata alone with parquet metadata (#7813)
* cache decoded region metadata Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix: account for decoded sst metadata cache weight * take optional pre-exist metadata Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
922f9cb3d6 |
feat: use dyn filter (#7545)
* parent
|
||
|
discord9
|
56ee8baa3f |
feat: admin gc table/regions (#7619)
* feat: gc table Signed-off-by: discord9 <discord9@163.com> * test: admin gc Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * refactor: GcStats Signed-off-by: discord9 <discord9@163.com> * refactor: use gc ticker for admin gc Signed-off-by: discord9 <discord9@163.com> * fix: region routes override Signed-off-by: discord9 <discord9@163.com> * test: non happy path Signed-off-by: discord9 <discord9@163.com> * refactor: gc job report enum Signed-off-by: discord9 <discord9@163.com> * test: process 0 regions Signed-off-by: discord9 <discord9@163.com> * after rebase Signed-off-by: discord9 <discord9@163.com> * feat: allow manual gc to return error Signed-off-by: discord9 <discord9@163.com> * chore: update proto Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: timeout and update proto Signed-off-by: discord9 <discord9@163.com> * chore: udpate proto Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Yingwen
|
0c30bf1a10 |
feat: add a subcommand to bench scan (#7722)
* feat: support scan bench Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support projection by name Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support force flat format Signed-off-by: evenyag <realevenyag@gmail.com> * feat: spawn tasks to poll streams Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support filter config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: scan bench support wal Signed-off-by: evenyag <realevenyag@gmail.com> * chore: support not providing provider in wal Signed-off-by: evenyag <realevenyag@gmail.com> * fix: skip wal replay Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: wrap EngineComponents Signed-off-by: evenyag <realevenyag@gmail.com> * docs: add scanbench doc Signed-off-by: evenyag <realevenyag@gmail.com> * chore: change --skip-wal-replay to --enable-wal Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove limit from config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
df04267c54 |
fix(repartition): reject writes on deallocating regions during region merge (#7694)
* feat(meta): add write route policy to region route with backward compatibility Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): use partition_expr compatibility accessor in repartition matching Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): introduce staging partition rule enum for repartition instructions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(datanode): plumb staging partition rule enum through heartbeat handlers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): mark pending-deallocate regions as reject-all during merge staging Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(partition): exclude reject-all regions from write partitioning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): store staging partition rule enum in region state Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): reject writes in staging when partition rule is reject-all Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): send enter staging instruction with reject-all Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(repartition): preserve reject-all on exit, merge enter-staging instructions, and allow staged bulk writes Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refactor to ignore all writes Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename StagingPartitionRule to StagingPartitionDirective across staging flow Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: nit Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
42ad842434 |
feat: support changing table's append_mode to true (#7669)
* feat: support alter append_mode to true Signed-off-by: evenyag <realevenyag@gmail.com> * test: add sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * fix: clear merge mode in mito when setting append mode Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize open request and options with both append/merge mode Signed-off-by: evenyag <realevenyag@gmail.com> * feat: clear merge mode when append mode is true Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
0ed3b83099 |
refactor: rename partition rule version to partition expr version (#7696)
* refactor: rename partition rule version to partition expr version Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: clippy Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |