mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-19 18:32:33 +00:00
346bbaba0efb1d50761948d3fd2408b0dbe40435
1071
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dhruv Vaishnav
|
7e2a75f771 |
feat: allow re-enabling WAL after disabling (#9130)
* feat: allow re-enabling WAL after disabling Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: regenerate skip_wal sqlness result The previous commit changed tests/cases/standalone/common/skip_wal.sql without regenerating the matching .result, so every Sqlness suite failed on the mismatch. tests/cases/distributed/common is a symlink to standalone/common, so the single stale file accounted for all five failing variants. Regenerated from a real run. The recorded output now covers the cases the .sql added: * A table created with skip_wal = 'true' has no real WAL provider, so enabling WAL is refused with Unsupported rather than the previous InvalidArguments. * A RaftEngine-backed table accepts the true -> false transition, and repeating it is a no-op. * SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including after a restart. * A row written while WAL was skipped is absent after restart, while a row written after WAL was restored survives. Verified locally with nothing else running on the machine: sqlness skip_wal passes in both the standalone and distributed environments, and the regenerated file is byte-identical to the output of an independent earlier run. Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: strengthen skip_wal provider coverage Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Fix WAL provider handling during re-enable Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Refactor WAL provider check Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
 Lei, HUANG
|
7aacc947b3 |
perf(mito2): lazily extract sparse primary key index values (#9176)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
1c01b541ec |
refactor(json2): JSON2 parquet projection and schema alignment (#9137)
* refactor(mito2): make JSON schema alignment targets explicit Signed-off-by: fys <fengys1996@gmail.com> * fix: cr * fix: cargo fmt --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
  discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
 Yingwen
|
7c85798ad0 |
feat(mito2): reconcile series indexes in background (#9086)
* feat(mito2): reconcile series indexes in background Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clean up series indexes published during region drop Signed-off-by: evenyag <realevenyag@gmail.com> * fix(config): align series index examples with upstream enable flag Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): run series index tasks on compaction runtime Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Weny Xu
|
d9b97796a6 |
fix: preserve count correctness after repartition (#9154)
* fix: preserve count correctness after repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: restore full predicate guard for count statistics Signed-off-by: WenyXu <wenymedia@gmail.com> * test: update count plans for full predicate guard Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve count statistics for safe partition scans Signed-off-by: WenyXu <wenymedia@gmail.com> * test: retain repartition home guard and cover staging flush Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
e0552eee2d |
fix(mito2): preserve effective sequences during compaction (#9147)
* fix(mito2): preserve effective row sequences during compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): inherit compaction input sequence bounds Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
ba0f7acd93 |
feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): correct float encoding checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover float SST encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile float encoding tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): release parquet test writer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): register float test primary key Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify BSS write lifecycles Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(metric-engine): verify BSS physical SST Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify bulk BSS lifecycle Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile bulk BSS test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding constructors Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float upgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float downgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding builder Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add default versus BSS storage comparison Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): align BSS reader benchmarks with prior study Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): parse current read benchmark averages Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): retain default float encoding in direct SST fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): isolate BSS user SSTs and benchmark every file Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): record measured BSS storage and reader tradeoffs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): expose warm scan variability and evidence limits Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): clarify BSS baseline and storage measurement scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): model bounded mixed integer and fractional metric series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): report bounded mixed BSS measurements and query regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): qualify timings affected by concurrent host builds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): include float BSS comparison in default regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): omit unsupported float encoding option from baseline setup Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
03d4e62a22 |
fix(mito2): deliver complete WAL entries without waiting for next input (#8916)
* fix(mito2): deliver complete WAL entries without waiting for next input Entry completeness is self-contained via Entry::is_complete(); no lookahead to the next entry is needed. Decode and yield complete entries immediately in both the log-store reader and the entry distributor. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): simplify live WAL entry fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
737025760e |
feat: support request-level WAL skipping for bulk inserts (#9110)
* feat: support request-level WAL skipping for bulk inserts Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover bulk insert WAL skipping across protocols Signed-off-by: WenyXu <wenymedia@gmail.com> * test: align WAL snapshot naming with sequence watermarks Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
23a1a5ecaa |
refactor: remove constant vector and replicate operation (#8999)
* refactor: remove constant vector Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove vector replicate operation Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve scalar vector types with optional type hints Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove obsolete mutable vector helper and import Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve typed nulls in struct scalar conversion Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
a23fe1c2dc |
fix: configure series indexes with an enable flag (#9141)
* fix: use join_dir for series index config path Signed-off-by: evenyag <realevenyag@gmail.com> * fix: configure series indexes with an enable flag Signed-off-by: evenyag <realevenyag@gmail.com> * docs: omit experimental series index from example configs Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve legacy cache cleanup path behavior Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove trivial path joining tests Signed-off-by: evenyag <realevenyag@gmail.com> * test: isolate worker group WAL directories on Windows Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
743261f05e |
fix(mito2): prevent JSON2 SWCS data loss from misaligned Parquet statistics due to projection (#9129)
* fix(mito2): look up row group stats by parquet leaf index for nested columns On flat-format tables a logical column can expand to multiple parquet leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per promoted path). ParquetFlat used the logical column index in the SST schema directly as the leaf index when reading row group statistics, so min/max/null stats of every column after a nested column were read from wrong leaves. When the misplaced leaf held order-compatible statistics (e.g. a small Int64 JSON path vs. the timestamp window predicate), min-max pruning dropped whole row groups by mistake. SWCS compaction reads inputs with a time window predicate, so it silently lost all rows of such files; plain queries with time-range predicates were affected as well. Map each column to its first parquet leaf column and report NoStats for columns with multiple leaves, which makes pruning conservative for them. Add a unit test and a sqlness regression case that reproduces the data loss on the unfixed binary. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): skip nested root stats and correct SWCS regression baseline Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve statistics leaves for primary-key SST readers Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): distinguish known null counts from unknown statistics Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): check JSON2 time pruning before SWCS compaction Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): validate all business columns after JSON2 SWCS Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
9696882217 |
fix(mito): preserve mixed JSON2 types during compaction (#9135)
* fix(mito): preserve mixed JSON2 types during compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito): assert all restored JSON2 rows in unordered merge test The regression test for aligning JSON2 layouts across unordered bulk parts only round-tripped the first part's `a` values. A merge that dropped or corrupted the second source's `b` values would still pass. Assert the merged batch has four rows and that the restored values of the second part (`{"b": 3}`, `{"b": 4}`) survive the merge, so the test covers both opaque sources as its comment claims. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
97648525cf |
perf(mito2): skip proven all-match prefilters (#9066)
* perf(mito2): skip proven all-match prefilters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add manual all-match prefilter reproduction Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover all-match prefilter execution paths Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): match prefilter fixture to sparse SST schema Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address all-match prefilter lint findings Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover all-match prefilters in default regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
b8e0f8e62e |
feat(mito2): add series index planning and builders (#9085)
* feat(mito2): add series index planning and builders Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): track window coverage and separate index builds Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): normalize index inputs and bound SST window expansion Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): simplify series index source summaries Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): assign series index deduplication TODO Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): initialize skip_wal in series index tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
7cf84892d2 |
perf(table): filter decoded rows with dynamic predicates (#9004)
* perf(table): filter decoded rows with dynamic predicates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve unknown dynamic filter rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(table): use null guards for dynamic filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): preserve null inputs during dynamic pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover frontend join dynamic filter transfer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(table): reset dynamic filters and scanner pruning state Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
c65a4e545e |
feat: make the series index resilient to time index unit widening (#8996)
* feat: record time unit per series index file The series index stores __series_min_ts/__series_max_ts as raw i64 in the time index unit at write time, but the searcher built its range predicates from the region's current unit, so files written before a time index unit widening would be compared in the wrong unit. Record the unit in the min/max ts fields' Arrow metadata when writing and build the per-file time predicates from it when searching, so each file is interpreted in the unit it was written with. The writer now also requires a timestamp time index. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on series index time units - split schema validation (validate_index_schema) from unit extraction (index_time_unit), distinguishing missing vs unsupported unit metadata in the errors instead of one misleading 'missing a valid metadata' - encode the recorded unit with an explicit exhaustive match rather than Debug formatting, so the on-disk encoding is reviewed next to its parser - extract time_index_unit to drop the unwrap in series_index_schema and the duplicated timestamp-time-index ensure in validate_metadata - test one searcher reading files with different recorded units, and the rejection of missing, unknown and mismatched units Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: reuse TimeUnit's Display form for the recorded unit string common_time::timestamp::TimeUnit already implements Display with the exact strings the series index records ("Second"/"Millisecond"/ "Microsecond"/"Nanosecond"), so drop the local time_unit_as_str mapping and use it; the parse side stays local since no FromStr counterpart exists anywhere yet. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit from its Display form in common-time Add FromStr for common_time::timestamp::TimeUnit, accepting exactly the Display form ("Second"/"Millisecond"/"Microsecond"/"Nanosecond") and failing with a new UnsupportedTimeUnit error (InvalidArguments). The series index now records and parses the unit with the common codec, dropping its local parse_time_unit. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: parse TimeUnit case-insensitively Lowercase the input before matching so "millisecond" and "MILLISECOND" parse like "Millisecond"; the error still reports the original string. Signed-off-by: Ning Sun <sunning@greptime.com> * feat: store series index min/max timestamps as native Timestamp columns Replace the Int64 min/max columns plus 'time_unit' field metadata with native Timestamp(unit) columns, so the unit rides on the datatype and each file is interpreted in the unit it was written with naturally. - series_index_schema types the columns from the time index unit; the writer reinterprets the raw i64 series bounds in that type (arrow's Int64->Timestamp cast reinterprets, it does not rescale) - the searcher reads the unit from each file's column datatype, builds Timestamp-typed predicates via datatypes' timestamp_to_scalar_value, and reinterprets parquet INT64 statistics in the column type so row-group pruning compares like-typed values - index files whose min/max columns are not Timestamp (written before this change) are rejected - the pruning test now asserts time-range predicates prune row groups, not just tag predicates Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop the TimeUnit string codec from common-time With the unit carried by the Timestamp datatype, the FromStr impl and UnsupportedTimeUnit error added for the field-metadata approach have no consumer; remove them. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: fold index file validation into a single pass The series index format is unreleased and unwired, so no compatibility classes are needed: validate_index_schema checks all columns and returns the min/max columns' unit directly, replacing the separate index_time_unit extraction. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: carry timestamps through SeriesIndexRow SeriesIndexRow and the aggregation path now hold common_time::Timestamp instead of raw i64s: timestamp_values interprets the input column in the writer's unit (rejecting a timestamp array whose unit differs, instead of silently reinterpreting it), and rows_to_batch builds the native Timestamp columns directly from the rows' units without an Int64 round trip through arrow cast. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: require a timestamp time index column in series index input The writer already refuses non-timestamp time indexes on the metadata side, and its input batches always carry the region's ts column as a timestamp array, so accepting plain Int64 columns only left a silent unit-interpretation hole; reject them instead. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: rescale series index input timestamps into the file unit The input array's unit is self-describing, so converting with Timestamp::convert_to cannot mislabel values; a mismatch no longer needs to be an error. Only a value that overflows the file's unit fails the write. This also makes the writer ready to aggregate old-unit batches after a time index widening. Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: drop redundant unit checks in series index writer The alter path flushes memtables before widening the region's time index unit, so a writer never receives batches in the region's previous unit. Reject a unit mismatch at the input boundary instead of rescaling per value, and build the index batch in the writer's recorded unit instead of re-deriving it from the schema and re-checking every row. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Yingwen
|
6fb5d2ebad |
feat(mito2): add series index catalog and lifecycle foundation (#9053)
* refactor(mito2): add series index catalog and lifecycle components Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): restore series index catalogs on region open Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify series index foundation and maintenance Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): separate series index purge task and simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> * docs: defer experimental series index configuration examples Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): make series index maintenance interval configurable Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): correct series index cleanup on close and drop Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): revert drop test changes Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation for series index settings Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use tokio unbounded channel for series index purger Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): simplify review test scope and clarify index config Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Lei, HUANGandCopilot Autofix powered by AI
|
be482553b0 |
fix(mito2): harden flat merge and use winner_tree (#9064)
* fix(mito2): reject non-flat batches in flat merge instead of panicking - SortColumns::new becomes fallible try_new: batches missing the flat-format internal columns (time index, __primary_key, __sequence) at the fixed trailing positions now yield InvalidRecordBatch instead of a downcast panic, completing the generic-schema gate that only covered BatchBuilder output assembly. Document the flat-format input contract on FlatMergeIterator/FlatMergeReader. - Clarify why BatchBuilder's schema gate uses >= 3 columns when a real flat-format schema always has at least 4. - Add schema-structure tests: empty primary keys (tables without tags), dictionary-encoded string tag columns with per-source dictionaries, and graceful rejection of batches without internal columns. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test(mito2): scan tables with various schemas through flat merge Add an engine-level test that writes, flushes and scans regions without tags (empty primary key) and with multiple string tags (dictionary-encoded in the flat input schema), so the flat merge reader merges an SST with the memtable on real schemas instead of hand-built batches. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): use winner_tree dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update comments for FlatMergeIterator struct Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Lei, HUANG
|
307fe0a692 |
feat(mito2): adapt bulk memtable encode threshold to write buffer size (#9056)
* feat(mito2): adapt bulk memtable encode bytes threshold to write buffer size The default encode_bytes_threshold is now max(64MB, min(global_write_buffer_size / 32, 512MB)) instead of a fixed 64MB, so it scales with the memtable budget. GREPTIME_BULK_ENCODE_BYTES_THRESHOLD and the per-region option still override the default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): clarify binary units in bulk threshold test The threshold test uses powers of 1024, so label its values as MiB and GiB instead of decimal MB and GB. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): resolve bulk encode threshold in memtable provider Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 XuanwoandWenyXu
|
4c12ea1aba |
chore(deps): bump opendal to 0.58.1 (#8742)
* chore(deps): bump opendal to 0.58.1 Upgrade direct opendal dependency and workspace object_store_opendal pin from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal 0.58.0). Adapt to OpenDAL 0.58 composition API: - Operator::new returns a finished operator; drop .finish() call sites - Replace HttpClientLayer / raw::HttpClient with OperationContext + HttpTransporter (ReqwestTransport) - Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to Service + Layer::apply_service - Rewrite SecureFs reader/writer/lister for sync factories and StreamRead - Use OperatorInfo::capability() instead of removed native_capability() Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: retrigger CI after udeps runner segfault Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(object-store): restore suffix read simulation for secure fs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: adapt remaining callers to OpenDAL 0.58 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
c3ea022de5 |
perf(mito2): blazing-fast tournament tree merger (#8989)
* perf(mito2): optimize flat merge heap and primary-key interleave Replace the per-row BinaryHeap pop/push cycle in FlatMerge with an in-place root mutation plus a single sift-down repair on a custom RootHeap, keeping the cold heap and direct-batch fast path unchanged. Fallible or awaiting batch transitions move the hot node out of the heap first, preserving error and cancellation semantics. Exploit the globally sorted merge output to build the internal Dictionary<UInt32, Binary> primary-key column with a one-pass ordered gather: append a Binary value only when the PK changes and reuse the current key for adjacent equal PKs, bypassing Arrow dictionary masks, hash interning and key remapping. Non-PK columns still use Arrow interleave. Also cache the current primary-key byte range in RowCursor to avoid repeated dictionary range decoding during comparisons, and add a setup-free Criterion benchmark with exact output-row assertions. 32-way/1 row-per-series/40-tag improves 955.79ms -> 562.36ms (-41.2%); 0-tag -39.5%, 64 rows/series -79.8%, 8-way -56.1%, single-iterator control +0.3%. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add rows-per-series sweep to flat merge bench Add 32-way/40-tag shapes for 1, 10, 100, 1000 and 10000 rows per series, and allow FLAT_MERGE_BENCH_SHAPE to match shape name prefixes so the whole sweep can run in one invocation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add oracle-based correctness tests for RootHeap Drive RootHeap and a std BinaryHeap oracle with the same seeded op sequence (push / pop / mutate-root + repair) and assert peek, len, best_child and the full drain order after every operation. A second run with a tiny value range makes duplicates dominate, covering the equal-key branches of sift_up/sift_down and best_child. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): replace hot heap with a tournament tree in flat merge Replace the hot RootHeap with a fixed-capacity tournament (winner) tree over per-node slots: every internal node caches the champion of its subtree, so advancing the winner only replays the ~log2(k) nodes on its leaf-to-root path with one compare per level, instead of the heap's two-compares-per-level sift that also re-compares the same node pairs on every row. Two fast paths keep dense shapes at O(1) per row: - champion retention: after mutating the winner in place, skip the replay entirely when it still beats the runner-up (its path caches are unchanged by construction); - a second-best slot cache, invalidated on any structural change, so the retention check costs a single compare without walking the tree. The cold heap, hot/cold overlap window, direct-batch fast path and the remove-before-fallible-fetch batch transition semantics are unchanged. Vs the RootHeap version: 1rps/32way/40tag -19.7%, 0tag -34.4%, 8way -15.9%, 64rps -30.5%, sweep 10/100/1000/10000rps -29~32%; vs the original BinaryHeap baseline the main shape is -52.8%. The single-iterator control is +8% (+50ns one-time construction allocation, no merge work). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): support generic schemas in flat merge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): satisfy clippy in flat merge benchmark Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): cache flat merge primary key index Compute the internal primary-key column index once when constructing BatchBuilder and reuse it for every output batch. Preserve the column-name gate for generic schemas. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark high-fan-in flat merges Add sparse 64, 128, 256, and 512-way merge shapes while keeping the total input fixed at 3.2 million rows. Compared with the merge-base heap implementation, median time improves by 56.0%, 60.8%, 55.6%, and 56.1%, respectively. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 LFC
|
d67d3501a9 |
fix(json2): keep empty structs in remainder (#9027)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
cf9a9639b0 |
perf(mito2): postpone covered time index filters (#8998)
* perf(mito2): postpone covered time index filters Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito2): reuse implied time range for prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): build finalized scan inputs Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): reject empty implied time filters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): guard last row shortcut with remaining filters Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
d7571c1278 |
feat(mito2): add SST range index searcher (#9003)
* feat(mito2): add SST range index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): reuse parquet index reader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify range index pruning Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): cover missing range index series Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
d351b7d471 |
feat(mito2): pass operation type to write cache upload hook (#9012)
Extend `WriteCacheUploadStoreWrapper::wrap` with the `OperationType` of the upload so implementations can apply per-operation policies (e.g. throttling compaction uploads but not flush uploads). Flush and compaction paths forward their existing `SstWriteRequest::op_type`; `put_and_upload_sst` is flush-only and index rebuild uploads are reported as compaction uploads. Files: `src/mito2/src/cache/write_cache.rs`, `src/mito2/src/sst/index.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
6f1dd0cb76 |
feat: allow widening the time index column's timestamp unit via ALTER TABLE, mito2 table only (#8894)
* feat: allow widening the time index column's timestamp unit via ALTER TABLE ... MODIFY COLUMN Previously MODIFY COLUMN rejected the time index column outright. Now the time index unit can be widened (Second -> Milli -> Micro -> Nano), which is lossless for data that fits the target unit: historical data in old SSTs is cast to the new unit on read by the existing schema-compat layer, and compaction rewrites it lazily. Narrowing and non-timestamp targets remain rejected; tag columns keep being rejected. Widening is rejected if any SST's time range would overflow the target unit's i64 range (e.g. millisecond -> nanosecond beyond year 2262), since the cast would silently null those values. Read-path correctness for old-unit SSTs (verified by new engine e2e tests and sqlness WHERE queries): - row-group min/max pruning: parquet statistics of a timestamp column are raw integers in the file's unit; when the region metadata's type differs (also the case for altered field columns), stats are now interpreted in the file's type and converted to the expected type before pruning. Without this, a new-unit predicate silently pruned whole row groups of old-unit files (wrong results, rows missing). - SST-level simple filters are skipped for columns whose file type differs from the expected type; the predicate is applied by the query layer's residual filter above the region scan. Also drop the stale "timestamp columns cannot change type" debug_assert. - retry idempotency: a same-type ModifyColumnType on the time index validates as a no-op and `need_alter` returns false, so a retried alter procedure (region already altered before the previous attempt failed) converges instead of aborting forever. - add TimeUnit ordering and ConcreteDataType::is_timestamp_unit_widening_to - relax ModifyColumnType validation in store-api and table metadata - tests: unit tests in datatypes/store-api/table; mito2 engine e2e tests (flushed SST + new writes + reopen + retry + overflow + predicate scans, cross-unit dedup, mixed-unit SSTs, compaction); sqlness cases incl. partitioned table and WHERE filters over old-unit data Signed-off-by: Ning Sun <sunning@greptime.com> * fix: resolve parquet filter issue * test: provide sqlness tests * refactor: drop trivial test cases and shorten comments Review pass over the branch's additions: - datatypes: keep a representative subset of the widening-matrix asserts - table: collapse the three single-branch rejection blocks into one loop - mito2: drop the boundary gt_eq and post-compaction predicate asserts (covered by the exact-filter regression test and sqlness); drop the engine-level gt_eq/lt_eq casts (full operator matrix stays in the cast_timestamp_unit unit tests) - sqlness: drop a bare full scan already covered by the filter above it - shorten function doc comments across datatypes/store-api/table/mito2/ recordbatch to the essential semantics Signed-off-by: Ning Sun <sunning@greptime.com> * fix: drop physical prefilter for columns whose file type differs Follow-up to the review feedback on CompatBatch/prune reader/filter handling for widened time index units. Between/InList/IsNull predicates are prefiltered by PhysicalFilterContext, which builds its physical expression against the FILE's schema while the predicate literals are in the expected (post-alter) unit. Evaluating them against an old-unit SST raised a cross-unit comparison error (Timestamp(ms) >= Timestamp(µs)) that failed the whole scan. Physical prefilter predicates are best-effort pruning hints (the query layer re-applies them above the scan), so drop the prefilter when the column's file type differs from the expected type, mirroring the simple-filter strategy. Verified: Between and a non-rewritten (large) InList on old-unit data no longer error and filter exactly end-to-end (sqlness), and no matching rows are lost at the engine level (engine test). Signed-off-by: Ning Sun <sunning@greptime.com> * test: add direct unit tests for stats cast and prefilter drop The two-step stats cast (reinterpret raw Int64 stats in the file's timestamp type, then rescale to the expected type) and the physical prefilter drop on file/expected type mismatch were only covered end-to-end; add localized unit tests so a regression fails at the exact site: - stats.rs (previously no tests): RowGroupPruningStats min/max over a hand-built RowGroupMetaData — passthrough with no expected metadata, passthrough on same type, and rescale (1000ms -> 1_000_000us, not 1000us) on a widened expected unit - reader.rs: PhysicalFilterContext::new_opt keeps a Between prefilter when file and expected types match and drops it on unit mismatch Signed-off-by: Ning Sun <sunning@greptime.com> * fix: tolerate mixed time units range cache key coverage check * docs: flag mixed-unit hazard in the (unwired) series index The series index stores per-series min/max ts as raw Int64 in the unit of the region metadata at write time, and the searcher builds its range predicates from a single per-region metadata. After a time index unit widen, files of one region would carry mixed units, so a per-file unit (or an index rebuild on such alters) is required before this index is wired into scans. Leave notes at both sites. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover mixed-unit compaction for sparse encoding and strict windows Compaction-path audit follow-up. The compat cast and window math were already covered for dense regions; add the two remaining e2e scenarios: - sparse primary key encoding (used by metric-engine physical regions): widening then compacting mixed-unit files rewrites the old-unit time index correctly through the sparse compaction compat path - strict-window manual compaction: each window output trims rows with a predicate built in the region's new unit against an old-unit file; every instant must survive exactly once (no loss, no cross-window duplication), rescaled Also documents the audit finding that Regular ranged (manual) compaction never trims rows: TwcsPicker sets output_time_range to None and the request time range only selects candidate windows. Signed-off-by: Ning Sun <sunning@greptime.com> * test: cover time index unit change in FlatCompatBatch directly The compat layer's rescaling of a widened time index was only verified end-to-end; add direct unit tests for both paths: - dense: identical units skip compat entirely; a widened unit rescales the time index column (1000ms -> 1_000_000us, not reinterpreted) while other columns pass through and the output schema matches the expected metadata - compact sparse (the metric-engine compaction path): same rescaling Signed-off-by: Ning Sun <sunning@greptime.com> * refactor: move timestamp unit division into common-time The exact unit division (UnitQuotient + div_mod_units) is time semantics, not filter logic; move it next to TimeUnit in common-time with a compact test covering representable/non-representable values, negative (floor) instants, and quotient overflow. The ScalarValue helpers stay in filter.rs since common-time has no datafusion dependency. Signed-off-by: Ning Sun <sunning@greptime.com> * test: compat rescales a widened time index and fills an added column together The realistic multi-alter sequence (widen at T1, add column at T2, read a T0 SST) exercises cast and default-fill in the same compute_index_and_fields pass; assert both in one output batch. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preflight time index widening overflow before any region alters Address review feedback on the overflow guard: - preflight: when the frontend operator receives a widening alter on the time index, run an existence scan (ts outside the target unit's i64 range, LIMIT 1, via the query engine so it covers every region of the table in both standalone and distributed modes) BEFORE any DDL task is submitted. A region that fits can no longer commit the new schema while another region rejects the alter with a non-retryable error. File and row-group pruning keep the scan cheap when nothing overflows. The per-region check in mito2 stays as the final guard for data written after the preflight (the remaining race window); without a validate-only wire field (region.proto lives in the external greptime-proto repo) a fully atomic two-phase validate/commit is out of scope here. - fast path: cast_timestamp_unit returns the filter unchanged when the literal is already in the target unit, skipping the div-mod rebuild. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments * fix: address auto review comments * fix: remove time index widening overflow preflight Overflow needs timestamps beyond the target unit's i64 range (~year 2262 for nanoseconds), which real workloads never write, so the two existence scans before every widening alter are not worth the cost. Region validation already rejects the alter when an SST's time range overflows the target unit; it now logs the rejection (with the offending file) and returns a deterministic client-facing message. Signed-off-by: Ning Sun <sunning@greptime.com> * test: make sqlness test stable * fix: log instead of rejecting time index widening overflow Overflowing values cast to NULL on read but do not otherwise affect reads or writes, so the alter is allowed; the region-level check now only logs (with the offending file) when an SST's time range exceeds the target unit's i64 range. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Lei, HUANG
|
f5212d3631 |
feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
27165c2fdc |
feat(mito2): add SST range index writer (#8954)
* feat(mito2): add SST range index writer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): share parquet index writer Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): lazily construct index writer context Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
6ce5747e02 |
fix(mito2): avoid chained L1 rewrites in TWCS (#8981)
* fix(mito2): avoid chained L1 rewrites in TWCS Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): fall back after ineligible L0 pick Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
9cfbc42126 |
refactor(mito2): prioritize file count in TWCS picker (#8765)
* refactor(mito2): prioritize file count in TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): migrate TWCS tests to async picker API Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): balance TWCS picks by file group Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first Previously pick_count_first filtered out multi-SST file groups and could pick singleton groups across them in one interval. Now multi-SST groups split the candidates into independent segments, so a single pick never crosses a multi-SST group. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant overlaps_files_left_behind check The rebase onto main kept both the upstream fix (#8872) and the branch's selected_overlaps_unselected check for the same deletion-marker problem. The upstream check is fully subsumed: files_to_merge only differs from the window files in append mode, where filter_deleted is already false, and selected_overlaps_unselected treats partially-selected groups as unselected, making it strictly stronger. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): pre-filter by selection time span in selected_overlaps_unselected A window group outside the overall time span of the selected groups cannot overlap any of them, so skip the precise overlap check (and the per-group file id set lookup) for it. In typical TWCS windows the pick is clustered in one segment, so most unselected groups are rejected in O(1). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove FileGroup abstraction from TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): drain compactable backlog after successful compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): score TWCS candidates by predicted progress Redefine what a pick candidate is worth: an interval is only eligible if compacting it makes progress on at least one axis — it reduces the physical file count given the max_output_file_size split threshold (predicted output = ceil(input bytes / threshold)), or it resolves at least one overlap between sorted runs. A pure rewrite that achieves neither (e.g. 32 large balanced files whose output would split back into just as many SSTs) is skipped instead of burning I/O. The candidate metrics are accumulated in a Candidate struct as the interval expands, and the score ranks by predicted file reduction, then overlap participants, then smaller input bytes. With no output size limit the behavior degenerates to the previous count-first rule. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make TWCS input limit configurable Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): apply TWCS trigger at window level Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Yingwen
|
28e2b5aca2 |
feat: expose missing SST manifest fields (#8965)
* feat: expose missing SST manifest fields Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve SST manifest column ordinals Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
ba3c5a939e |
chore(mito2): reduce default auto flush interval (#8971)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
bd7d2c1dfa |
fix(mito2): use target sequence for foreign SSTs (#8946)
* fix(mito2): use target sequence for foreign SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address foreign SST review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
932f87f7a8 |
refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout - route missing JSON2 paths to the v2 remainder field - reconstruct complete values from explicit fields and remainder data - preserve root JSON2 columns across projections and filters - support nested JSON values in json_get string results - add and reorganize JSON2 sqlness coverage Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
a3a0db63b8 |
feat(mito2): add series index searcher (#8926)
* feat(mito2): add series index searcher Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use parquet push decoder for series index Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle evolved series index schemas Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANGandYingwen
|
4ac3423261 |
fix(mito2): split SSTs at primary key series boundaries (#8888)
* fix(mito2): split SSTs at series boundaries Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover SST splitting without primary key Also document the sortedness precondition and the series boundary split semantics on write_all_flat/write_all_flat_as_primary_key and the new split helpers. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): avoid per-row Vec allocation for empty primary key Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Update src/mito2/src/sst/parquet/writer.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
Weny Xu
|
a502dfdefd |
fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
4126cf99b6 |
refactor(mito2): rename pk index to series index (#8893)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
cb30837cd5 |
feat: add incremental primary key index writer (#8788)
* feat: initial implementation of the pk index writer Signed-off-by: evenyag <realevenyag@gmail.com> * perf: optimize primary key index writer Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add todo Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito2): track series count in pk index metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify pk index writer cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clean up aborted pk index writers Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
76924c2d36 |
feat(mito2): introduce two-phase metric series scans (#8826)
* feat(mito2): add two-phase series scan Signed-off-by: evenyag <realevenyag@gmail.com> * docs: regenerate configuration reference Signed-off-by: evenyag <realevenyag@gmail.com> * test(sqlness): update series scan explain results Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config API expectation Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): bound two-phase series discovery Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): avoid candidate distribution deadlock Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove obsolete dead code allowances Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): share series scan memory pool Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
fys
|
6538db6d61 |
refactor(json2): push down json2 type hints to parquet reads (#8833)
* refactor(mito): push down json2 type hints to parquet reads Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share json2 target types with arc Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): derive json2 output schema from target types Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): simplify read columns construction Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo check Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): reject JSON hints for non-JSON2 read columns Signed-off-by: fys <fengys1996@gmail.com> * fix: do not pushdown json type hint of non-json2-col Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> * refactor(query): simplify JSON type hint application Signed-off-by: fys <fengys1996@gmail.com> * refactor: clean up JSON2 type hint handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): keep JSON2 hints with flat read format Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): use raw parquet projection for output schema Signed-off-by: fys <fengys1996@gmail.com> * fix(query): note JSON2 hint scope limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): store JSON target types as native types Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover join hint qualifier limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): remove JSON2 fallback from compat cast Signed-off-by: fys <fengys1996@gmail.com> * docs(mito): document ReadColumns ordering contract Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo clippy Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
discord9
|
f692ac32f3 |
test(mito2): isolate sequence publication barrier (#8876)
* test(mito2): isolate sequence publication barrier Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim sequence barrier test noise Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 jeremyhi
|
3c80df043a |
fix(mito2): keep deletion markers when compacting part of a window (#8872)
`TwcsPicker::find_inputs` decides `filter_deleted` from the shape of the whole time window, but the compaction inputs are only a subset of it: `reduce_runs` and `merge_seq_files` narrow the selection down and the max input file num limit narrows it further. When a deletion marker lands in the compacted set while the file holding the row it masks stays behind, the marker is dropped from the output and the old row becomes visible again. Re-check the final selection against the rest of the window and stop filtering deleted rows whenever something left behind still overlaps the inputs. The check compares ranges inclusively, so it also covers files that share only a boundary timestamp: run detection treats those as non-overlapping, which is how a single timestamp delete file ends up in the same run as the file it deletes rows from. Signed-off-by: jeremyhi <fengjiachun@gmail.com> |