mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-27 22:26:15 +00:00
a6c3dd7bcc1dcd1dab68fdb09854664543a6fded
1068
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
      
|
393c68b101 |
fix: backport remaining small fixes to release/5e02efa9 (#9809)
* fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> (cherry picked from commit |
||
|
Lei, HUANG
|
fba23dad8e |
fix(mito2): backport primary key range comparison across schema versions to release/5e02efa9 (#9803)
* fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions
Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.
Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): report invalid primary key ranges
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): scope primary key ranges to comparisons
Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.
Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.
Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): cache aligned primary key ranges per file
Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.
Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(mito2): clarify primary key mapper schema snapshot
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): align primary key range fixtures with table contract
Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.
Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): enable filesystem GC in release unit tests
Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito-codec): skip release value decoding in PK prefix counts
Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.
Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito-codec): remove redundant prefix integration tests
Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: wait for MySQL to accept authenticated TCP queries
Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.
Verify readiness with SELECT 1 over TCP rather than the initialization socket.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit
|
||
 Lei, HUANGandYingwen
|
453f8ca888 |
fix: backport disabling v2 series scan by default to release/5e02efa9 (#9784)
fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily
* test: explicitly enable v2 series scan in sqlness configs
* test: regenerate sqlness results for legacy series scan default
---------
(cherry picked from commit
|
||
   
|
2b05da4c06 |
fix: backport suggested independent fixes to release/5e02efa9 (#9795)
* fix(servers): resolve port from IPv6 bind addresses (#9142) Signed-off-by: immanuwell <pchpr.00@list.ru> (cherry picked from commit |
||
|
|
fd3faeb7ed |
fix: backport scan statistics and partitioning fixes to release/5e02efa9 (#9783)
* fix: preserve count correctness after repartition (#9154) * fix: preserve count correctness after repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: restore full predicate guard for count statistics Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update count plans for full predicate guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve count statistics for safe partition scans Signed-off-by: WenyXu <wenymedia@gmail.com> * test: retain repartition home guard and cover staging flush Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
Lei, HUANGandYingwen
|
1e5fef5809 |
fix(mito): backport bulk memtable version ordering fix to release/5e02efa9 (#9789)
* fix(mito2): preserve last-non-null values across compaction (#9169) * fix(mito2): preserve last-non-null values during compaction Use a dedicated LastNonNullPicker to close TWCS seeds over overlapping SSTs, merge connected seed groups, and defer closures with busy inputs. Keep LastRow and strict-window selection unchanged. Index overlap candidates on the blocking compaction runtime and cover field promotion, transitive dependencies, expiration, and conservative key-range pruning with regression tests. Refs #9146 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard last-non-null compaction against pending memtables Track conservative memtable sequence lower bounds and capture them from the same version as compaction inputs. Defer unsafe SST closures and strict-window plans until pending writes no longer intersect their sequence span. Cache minimum sequences during bulk conversion and expose a lightweight memtable getter to avoid rescanning rows or computing full statistics. Fixes #9157 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): limit eligible last-non-null compaction outputs Enumerate all TWCS seeds for LastNonNull while retaining bounded window planning concurrency. Apply the output budget after closure expansion and busy/memtable safety checks, preserving execution priority and ordinary TWCS behavior. Cover rejected and absorbed seeds, priority ordering, and bounded and unlimited output counts with real-picker regression tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): include all candidates before last-non-null closure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify last-non-null compaction closure Extract transitive closure expansion from output scheduling and clarify seed handling. Document the dedicated picker rationale and schema-evolution risks in PK overlap checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover deferred non-seed compaction bridges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): defer strict-window compaction with busy dependencies Keep busy SSTs when building the LastNonNull strict-window dependency closure, then defer the entire rewrite plan if any selected input is busy. Preserve LastRow filtering and expired-file cleanup.\n\nCover requested, transitive and unrelated busy windows, including retry after the dependency is released. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document compaction overlap index fields Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit |
||
|
Lei, HUANG
|
1506974bd9 |
fix: backport SST write buffer release to release/5e02efa9 (#9774)
* fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: track removal of the OpenDAL bridge fork
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): honor SST write buffers across upload paths
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
(cherry picked from commit
|
||
|
508d1a1a33 |
fix: backport clean must-have fixes to release/5e02efa9 (#9772)
* fix(mito2): preserve effective sequences during compaction (#9147) * fix(mito2): preserve effective row sequences during compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): inherit compaction input sequence bounds Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit |
||
|
Lei, HUANGandWeny Xu
|
8282f72c74 |
fix(mito): backport skip-WAL close flush to release/5e02efa9 (#9625)
Track skip-WAL writes across mutable and immutable memtables and flush them on graceful region close. Distinguish incoming requests from WAL replay, and preserve WAL-only recovery tests by bypassing Close.
Adapt bulk tracking to the release branch's existing region-level WAL policy without adding request-level bulk skip-WAL support. Keep release-only tests and omit changes to the absent SST primary-key module.
(cherry picked from commit
|
||
|
Lei, HUANGandWeny Xu
|
12b7ebe7d5 |
feat: backport bulk skip-WAL support and integration coverage (#9627)
* test: backport request-level skip-WAL integration coverage Backport #9093 as the shared protocol and storage test harness prerequisite for bulk request WAL skipping. Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
Weny Xu
|
372f33540a |
feat(frontend): backport write retries to release/5e02efa9 (#9569)
* feat(client): support region request reuse and route refresh hints (#9540) * feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
|
dennis zhuang
|
c66a4aeb37 |
fix(mito2): hold the bulk compact permit for the whole blocking task (#9252)
Keep the semaphore permit alive throughout queued and running bulk compaction in `src/mito2/src/memtable/bulk.rs`.
(cherry picked from commit
|
||
|
Lei, HUANG
|
1cb529510c |
fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
Backport #9304 across `access_layer.rs`, `cache.rs`, `cache/file_cache.rs`, `read/pruner.rs`, `region/opener.rs`, `sst/index.rs`, and `sst/parquet/reader.rs` under `src/mito2/src`.
(cherry picked from commit
|
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
7cf84892d2 |
perf(table): filter decoded rows with dynamic predicates (#9004)
* perf(table): filter decoded rows with dynamic predicates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve unknown dynamic filter rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(table): use null guards for dynamic filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve null inputs during dynamic pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover frontend join dynamic filter transfer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): reset dynamic filters and scanner pruning state Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Ning Sun
|
c65a4e545e |
feat: make the series index resilient to time index unit widening (#8996)
* feat: record time unit per series index file The series index stores __series_min_ts/__series_max_ts as raw i64 in the time index unit at write time, but the searcher built its range predicates from the region's current unit, so files written before a time index unit widening would be compared in the wrong unit. Record the unit in the min/max ts fields' Arrow metadata when writing and build the per-file time predicates from it when searching, so each file is interpreted in the unit it was written with. The writer now also requires a timestamp time index. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on series index time units - split schema validation (validate_index_schema) from unit extraction (index_time_unit), distinguishing missing vs unsupported unit metadata in the errors instead of one misleading 'missing a valid metadata' - encode the recorded unit with an explicit exhaustive match rather than Debug formatting, so the on-disk encoding is reviewed next to its parser - extract time_index_unit to drop the unwrap in series_index_schema and the duplicated timestamp-time-index ensure in validate_metadata - test one searcher reading files with different recorded units, and the rejection of missing, unknown and mismatched units Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: reuse TimeUnit's Display form for the recorded unit string common_time::timestamp::TimeUnit already implements Display with the exact strings the series index records ("Second"/"Millisecond"/ "Microsecond"/"Nanosecond"), so drop the local time_unit_as_str mapping and use it; the parse side stays local since no FromStr counterpart exists anywhere yet. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit from its Display form in common-time Add FromStr for common_time::timestamp::TimeUnit, accepting exactly the Display form ("Second"/"Millisecond"/"Microsecond"/"Nanosecond") and failing with a new UnsupportedTimeUnit error (InvalidArguments). The series index now records and parses the unit with the common codec, dropping its local parse_time_unit. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit case-insensitively Lowercase the input before matching so "millisecond" and "MILLISECOND" parse like "Millisecond"; the error still reports the original string. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: store series index min/max timestamps as native Timestamp columns Replace the Int64 min/max columns plus 'time_unit' field metadata with native Timestamp(unit) columns, so the unit rides on the datatype and each file is interpreted in the unit it was written with naturally. - series_index_schema types the columns from the time index unit; the writer reinterprets the raw i64 series bounds in that type (arrow's Int64->Timestamp cast reinterprets, it does not rescale) - the searcher reads the unit from each file's column datatype, builds Timestamp-typed predicates via datatypes' timestamp_to_scalar_value, and reinterprets parquet INT64 statistics in the column type so row-group pruning compares like-typed values - index files whose min/max columns are not Timestamp (written before this change) are rejected - the pruning test now asserts time-range predicates prune row groups, not just tag predicates Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop the TimeUnit string codec from common-time With the unit carried by the Timestamp datatype, the FromStr impl and UnsupportedTimeUnit error added for the field-metadata approach have no consumer; remove them. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: fold index file validation into a single pass The series index format is unreleased and unwired, so no compatibility classes are needed: validate_index_schema checks all columns and returns the min/max columns' unit directly, replacing the separate index_time_unit extraction. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: carry timestamps through SeriesIndexRow SeriesIndexRow and the aggregation path now hold common_time::Timestamp instead of raw i64s: timestamp_values interprets the input column in the writer's unit (rejecting a timestamp array whose unit differs, instead of silently reinterpreting it), and rows_to_batch builds the native Timestamp columns directly from the rows' units without an Int64 round trip through arrow cast. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: require a timestamp time index column in series index input The writer already refuses non-timestamp time indexes on the metadata side, and its input batches always carry the region's ts column as a timestamp array, so accepting plain Int64 columns only left a silent unit-interpretation hole; reject them instead. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: rescale series index input timestamps into the file unit The input array's unit is self-describing, so converting with Timestamp::convert_to cannot mislabel values; a mismatch no longer needs to be an error. Only a value that overflows the file's unit fails the write. This also makes the writer ready to aggregate old-unit batches after a time index widening. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop redundant unit checks in series index writer The alter path flushes memtables before widening the region's time index unit, so a writer never receives batches in the region's previous unit. Reject a unit mismatch at the input boundary instead of rescaling per value, and build the index batch in the writer's recorded unit instead of re-deriving it from the schema and re-checking every row. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Yingwen
|
6fb5d2ebad |
feat(mito2): add series index catalog and lifecycle foundation (#9053)
* refactor(mito2): add series index catalog and lifecycle components Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): restore series index catalogs on region open Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify series index foundation and maintenance Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): separate series index purge task and simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> * docs: defer experimental series index configuration examples Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): make series index maintenance interval configurable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): correct series index cleanup on close and drop Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): revert drop test changes Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation for series index settings Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use tokio unbounded channel for series index purger Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): simplify review test scope and clarify index config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANGandCopilot Autofix powered by AI
|
be482553b0 |
fix(mito2): harden flat merge and use winner_tree (#9064)
* fix(mito2): reject non-flat batches in flat merge instead of panicking - SortColumns::new becomes fallible try_new: batches missing the flat-format internal columns (time index, __primary_key, __sequence) at the fixed trailing positions now yield InvalidRecordBatch instead of a downcast panic, completing the generic-schema gate that only covered BatchBuilder output assembly. Document the flat-format input contract on FlatMergeIterator/FlatMergeReader. - Clarify why BatchBuilder's schema gate uses >= 3 columns when a real flat-format schema always has at least 4. - Add schema-structure tests: empty primary keys (tables without tags), dictionary-encoded string tag columns with per-source dictionaries, and graceful rejection of batches without internal columns. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test(mito2): scan tables with various schemas through flat merge Add an engine-level test that writes, flushes and scans regions without tags (empty primary key) and with multiple string tags (dictionary-encoded in the flat input schema), so the flat merge reader merges an SST with the memtable on real schemas instead of hand-built batches. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): use winner_tree dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update comments for FlatMergeIterator struct Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Lei, HUANG
|
307fe0a692 |
feat(mito2): adapt bulk memtable encode threshold to write buffer size (#9056)
* feat(mito2): adapt bulk memtable encode bytes threshold to write buffer size The default encode_bytes_threshold is now max(64MB, min(global_write_buffer_size / 32, 512MB)) instead of a fixed 64MB, so it scales with the memtable budget. GREPTIME_BULK_ENCODE_BYTES_THRESHOLD and the per-region option still override the default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): clarify binary units in bulk threshold test The threshold test uses powers of 1024, so label its values as MiB and GiB instead of decimal MB and GB. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): resolve bulk encode threshold in memtable provider Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 XuanwoandWenyXu
|
4c12ea1aba |
chore(deps): bump opendal to 0.58.1 (#8742)
* chore(deps): bump opendal to 0.58.1 Upgrade direct opendal dependency and workspace object_store_opendal pin from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal 0.58.0). Adapt to OpenDAL 0.58 composition API: - Operator::new returns a finished operator; drop .finish() call sites - Replace HttpClientLayer / raw::HttpClient with OperationContext + HttpTransporter (ReqwestTransport) - Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to Service + Layer::apply_service - Rewrite SecureFs reader/writer/lister for sync factories and StreamRead - Use OperatorInfo::capability() instead of removed native_capability() Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: retrigger CI after udeps runner segfault Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(object-store): restore suffix read simulation for secure fs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: adapt remaining callers to OpenDAL 0.58 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
c3ea022de5 |
perf(mito2): blazing-fast tournament tree merger (#8989)
* perf(mito2): optimize flat merge heap and primary-key interleave Replace the per-row BinaryHeap pop/push cycle in FlatMerge with an in-place root mutation plus a single sift-down repair on a custom RootHeap, keeping the cold heap and direct-batch fast path unchanged. Fallible or awaiting batch transitions move the hot node out of the heap first, preserving error and cancellation semantics. Exploit the globally sorted merge output to build the internal Dictionary<UInt32, Binary> primary-key column with a one-pass ordered gather: append a Binary value only when the PK changes and reuse the current key for adjacent equal PKs, bypassing Arrow dictionary masks, hash interning and key remapping. Non-PK columns still use Arrow interleave. Also cache the current primary-key byte range in RowCursor to avoid repeated dictionary range decoding during comparisons, and add a setup-free Criterion benchmark with exact output-row assertions. 32-way/1 row-per-series/40-tag improves 955.79ms -> 562.36ms (-41.2%); 0-tag -39.5%, 64 rows/series -79.8%, 8-way -56.1%, single-iterator control +0.3%. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add rows-per-series sweep to flat merge bench Add 32-way/40-tag shapes for 1, 10, 100, 1000 and 10000 rows per series, and allow FLAT_MERGE_BENCH_SHAPE to match shape name prefixes so the whole sweep can run in one invocation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add oracle-based correctness tests for RootHeap Drive RootHeap and a std BinaryHeap oracle with the same seeded op sequence (push / pop / mutate-root + repair) and assert peek, len, best_child and the full drain order after every operation. A second run with a tiny value range makes duplicates dominate, covering the equal-key branches of sift_up/sift_down and best_child. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): replace hot heap with a tournament tree in flat merge Replace the hot RootHeap with a fixed-capacity tournament (winner) tree over per-node slots: every internal node caches the champion of its subtree, so advancing the winner only replays the ~log2(k) nodes on its leaf-to-root path with one compare per level, instead of the heap's two-compares-per-level sift that also re-compares the same node pairs on every row. Two fast paths keep dense shapes at O(1) per row: - champion retention: after mutating the winner in place, skip the replay entirely when it still beats the runner-up (its path caches are unchanged by construction); - a second-best slot cache, invalidated on any structural change, so the retention check costs a single compare without walking the tree. The cold heap, hot/cold overlap window, direct-batch fast path and the remove-before-fallible-fetch batch transition semantics are unchanged. Vs the RootHeap version: 1rps/32way/40tag -19.7%, 0tag -34.4%, 8way -15.9%, 64rps -30.5%, sweep 10/100/1000/10000rps -29~32%; vs the original BinaryHeap baseline the main shape is -52.8%. The single-iterator control is +8% (+50ns one-time construction allocation, no merge work). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): support generic schemas in flat merge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): satisfy clippy in flat merge benchmark Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): cache flat merge primary key index Compute the internal primary-key column index once when constructing BatchBuilder and reuse it for every output batch. Preserve the column-name gate for generic schemas. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark high-fan-in flat merges Add sparse 64, 128, 256, and 512-way merge shapes while keeping the total input fixed at 3.2 million rows. Compared with the merge-base heap implementation, median time improves by 56.0%, 60.8%, 55.6%, and 56.1%, respectively. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 LFC
|
d67d3501a9 |
fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
cf9a9639b0 |
perf(mito2): postpone covered time index filters (#8998)
* perf(mito2): postpone covered time index filters Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito2): reuse implied time range for prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): build finalized scan inputs Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): reject empty implied time filters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): guard last row shortcut with remaining filters Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
d7571c1278 |
feat(mito2): add SST range index searcher (#9003)
* feat(mito2): add SST range index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse parquet index reader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify range index pruning Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover missing range index series Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
d351b7d471 |
feat(mito2): pass operation type to write cache upload hook (#9012)
Extend `WriteCacheUploadStoreWrapper::wrap` with the `OperationType` of the upload so implementations can apply per-operation policies (e.g. throttling compaction uploads but not flush uploads). Flush and compaction paths forward their existing `SstWriteRequest::op_type`; `put_and_upload_sst` is flush-only and index rebuild uploads are reported as compaction uploads. Files: `src/mito2/src/cache/write_cache.rs`, `src/mito2/src/sst/index.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Lei, HUANG
|
f5212d3631 |
feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
27165c2fdc |
feat(mito2): add SST range index writer (#8954)
* feat(mito2): add SST range index writer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): share parquet index writer Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): lazily construct index writer context Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
6ce5747e02 |
fix(mito2): avoid chained L1 rewrites in TWCS (#8981)
* fix(mito2): avoid chained L1 rewrites in TWCS Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): fall back after ineligible L0 pick Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
9cfbc42126 |
refactor(mito2): prioritize file count in TWCS picker (#8765)
* refactor(mito2): prioritize file count in TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): migrate TWCS tests to async picker API Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): balance TWCS picks by file group Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first Previously pick_count_first filtered out multi-SST file groups and could pick singleton groups across them in one interval. Now multi-SST groups split the candidates into independent segments, so a single pick never crosses a multi-SST group. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant overlaps_files_left_behind check The rebase onto main kept both the upstream fix (#8872) and the branch's selected_overlaps_unselected check for the same deletion-marker problem. The upstream check is fully subsumed: files_to_merge only differs from the window files in append mode, where filter_deleted is already false, and selected_overlaps_unselected treats partially-selected groups as unselected, making it strictly stronger. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): pre-filter by selection time span in selected_overlaps_unselected A window group outside the overall time span of the selected groups cannot overlap any of them, so skip the precise overlap check (and the per-group file id set lookup) for it. In typical TWCS windows the pick is clustered in one segment, so most unselected groups are rejected in O(1). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove FileGroup abstraction from TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): drain compactable backlog after successful compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): score TWCS candidates by predicted progress Redefine what a pick candidate is worth: an interval is only eligible if compacting it makes progress on at least one axis — it reduces the physical file count given the max_output_file_size split threshold (predicted output = ceil(input bytes / threshold)), or it resolves at least one overlap between sorted runs. A pure rewrite that achieves neither (e.g. 32 large balanced files whose output would split back into just as many SSTs) is skipped instead of burning I/O. The candidate metrics are accumulated in a Candidate struct as the interval expands, and the score ranks by predicted file reduction, then overlap participants, then smaller input bytes. With no output size limit the behavior degenerates to the previous count-first rule. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make TWCS input limit configurable Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): apply TWCS trigger at window level Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
28e2b5aca2 |
feat: expose missing SST manifest fields (#8965)
* feat: expose missing SST manifest fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve SST manifest column ordinals Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
ba3c5a939e |
chore(mito2): reduce default auto flush interval (#8971)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
bd7d2c1dfa |
fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address foreign SST review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
932f87f7a8 |
refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout - route missing JSON2 paths to the v2 remainder field - reconstruct complete values from explicit fields and remainder data - preserve root JSON2 columns across projections and filters - support nested JSON values in json_get string results - add and reorganize JSON2 sqlness coverage Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
a3a0db63b8 |
feat(mito2): add series index searcher (#8926)
* feat(mito2): add series index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use parquet push decoder for series index Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle evolved series index schemas Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANGandYingwen
|
4ac3423261 |
fix(mito2): split SSTs at primary key series boundaries (#8888)
* fix(mito2): split SSTs at series boundaries Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover SST splitting without primary key Also document the sortedness precondition and the series boundary split semantics on write_all_flat/write_all_flat_as_primary_key and the new split helpers. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): avoid per-row Vec allocation for empty primary key Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update src/mito2/src/sst/parquet/writer.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
Weny Xu
|
a502dfdefd |
fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
4126cf99b6 |
refactor(mito2): rename pk index to series index (#8893)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
cb30837cd5 |
feat: add incremental primary key index writer (#8788)
* feat: initial implementation of the pk index writer Signed-off-by: evenyag <realevenyag@gmail.com> * perf: optimize primary key index writer Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add todo Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): track series count in pk index metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify pk index writer cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clean up aborted pk index writers Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
76924c2d36 |
feat(mito2): introduce two-phase metric series scans (#8826)
* feat(mito2): add two-phase series scan Signed-off-by: evenyag <realevenyag@gmail.com> * docs: regenerate configuration reference Signed-off-by: evenyag <realevenyag@gmail.com> * test(sqlness): update series scan explain results Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): bound two-phase series discovery Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid candidate distribution deadlock Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove obsolete dead code allowances Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): share series scan memory pool Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 fys
|
6538db6d61 |
refactor(json2): push down json2 type hints to parquet reads (#8833)
* refactor(mito): push down json2 type hints to parquet reads Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share json2 target types with arc Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): derive json2 output schema from target types Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): simplify read columns construction Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo check Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): reject JSON hints for non-JSON2 read columns Signed-off-by: fys <fengys1996@gmail.com> * fix: do not pushdown json type hint of non-json2-col Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> * refactor(query): simplify JSON type hint application Signed-off-by: fys <fengys1996@gmail.com> * refactor: clean up JSON2 type hint handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): keep JSON2 hints with flat read format Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): use raw parquet projection for output schema Signed-off-by: fys <fengys1996@gmail.com> * fix(query): note JSON2 hint scope limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): store JSON target types as native types Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover join hint qualifier limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): remove JSON2 fallback from compat cast Signed-off-by: fys <fengys1996@gmail.com> * docs(mito): document ReadColumns ordering contract Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo clippy Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
f692ac32f3 |
test(mito2): isolate sequence publication barrier (#8876)
* test(mito2): isolate sequence publication barrier Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim sequence barrier test noise Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
3c80df043a |
fix(mito2): keep deletion markers when compacting part of a window (#8872)
`TwcsPicker::find_inputs` decides `filter_deleted` from the shape of the whole time window, but the compaction inputs are only a subset of it: `reduce_runs` and `merge_seq_files` narrow the selection down and the max input file num limit narrows it further. When a deletion marker lands in the compacted set while the file holding the row it masks stays behind, the marker is dropped from the output and the old row becomes visible again. Re-check the final selection against the rest of the window and stop filtering deleted rows whenever something left behind still overlaps the inputs. The check compares ranges inclusively, so it also covers files that share only a boundary timestamp: run detection treats those as non-overlapping, which is how a single timestamp delete file ends up in the same run as the file it deletes rows from. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
7539e60139 |
refactor: remove trivial tests (#8877)
* refactor: remove trivial tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: remove unused trace test import Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 Lanqing Yang
|
0e28916695 |
perf(mito2): optimize dictionary primary key sorting (#8767)
Signed-off-by: lyang24 <lanqingy93@gmail.com> |
||
|
discord9
|
d0fecdd6b0 |
fix(mito2): publish committed sequence only after rows are installed (#8862)
The committed-sequence watermark must never cover rows that are not yet physically visible. Previously write_memtable() published next_sequence - 1 before bulk parts were installed, so a scan opening a snapshot could bind H to invisible sequences and permanently miss rows after checkpoint advance. Publish once, after both ordinary and bulk memtable writes complete, in the single-region fast path, the multi-region spawned tasks, and WAL replay; skip publication for contexts whose WAL entry could not be built. Add a deterministic worker-level race test using a cfg(test) bulk-install barrier proving the committed sequence stays put until installation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |