* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows
Split the single TWCS trigger_file_num into per-window-state thresholds:
the active window keeps the existing trigger (default 4, legacy
compaction.twcs.trigger_file_num stays a compatible alias), while
inactive windows use a new trigger (default 2). Inactive windows
additionally fall back from balanced L0-only/L1-only candidates to a
progress-making unbalanced mixed candidate so historical windows can
converge; the active window retains the row/byte balance guards.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): bound inactive TWCS window convergence by rewrite budget
Inactive windows that cannot compact within one level previously either
stayed stuck (a threshold-qualified but unbalanced level returned no
candidate without trying any fallback) or fell back to a mixed merge
with no balance checks at all, which could rewrite a huge compacted file
to absorb tiny fresh files.
Inactive windows now converge progressively: threshold-qualified
balanced picks, sub-threshold balanced single-level picks, a mixed merge
whose total rewrite must fit in the output file budget, and finally an
L0-only merge without balance checks. Windows that qualify for none of
these are left uncompacted, bounding write amplification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(mito2): cover TWCS window trigger options in alter_table_options sqlness case
Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and
compaction.twcs.inactive_window.trigger_file_num end to end, including
that setting the canonical active key removes the legacy
compaction.twcs.trigger_file_num alias from the table options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): derive TWCS active window from the max-sequence file
The active window was determined by the max event-time window among
level-0 files. Between an L0 compaction removing its inputs and the next
flush landing, level 0 is empty, so the active window transiently became
None and every window fell back to the inactive rules - triggering
full-window convergence merges during ongoing ingestion whose outputs
are then superseded by new data.
Flush and compaction outputs both inherit the max input sequence, so the
file with the highest sequence across all levels always tracks the most
recent write. Use its window as the active window, falling back to the
previous L0-based rule when no file carries a sequence (legacy files).
The new helper deliberately computes window keys with the
assign_to_windows convention (truncate to seconds, then align up),
because the result is compared against window keys produced there; the
older ceil-based helper is kept unchanged for the legacy fallback path.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): add active-window L1 compaction safety trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(compat): cover TWCS active window options
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): align TWCS window trigger validation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): resolve database TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): prioritize newer compaction windows
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): repick serial compaction outputs
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): configure inactive-window L1 trigger
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): prioritize TWCS compaction candidates
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): preserve TWCS trigger downgrade compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): distinguish invalid database option values
Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(meta): explain TWCS legacy key compatibility
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify active window trigger field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): normalize TWCS trigger aliases
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): ignore ineligible files for active window
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): mark explicit TWCS options as overrides
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito2): normalize zero compaction output size
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(options): validate database TWCS trigger values
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* style(store-api): collapse TWCS alias conflict condition
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(mito2): support exact sequence range reads
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): cover preserve row sequence table alter
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): clear preserve_row_sequence marker on copy_region_from
copy_region_from copies source FileMeta into the target region, which has an
independent sequence domain. The physical per-row sequences in the copied
file belong to the source region only; trusting them in the target would let
an exact sequence-range request replay source-domain rows as if they were
target sequences. Clear the preserve_row_sequence marker on copied files so
the target fails closed with SequenceRangeUnsupported until the scan provably
cannot intersect the copied rows.
Add a regression test: copying from a preserve-enabled source into a
preserve-enabled target clears the marker, and an exact (2, 7] request on the
target returns SequenceRangeUnsupported instead of replaying source rows.
Fixes#8865
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style: remove redundant doc comments for exact sequence range options
Approved comment-cleanup-only changes for #8865: drop outdated doc
summaries duplicated on the exact_sequence_range wrapper and the
preserve_row_sequence field, drop pure-restatement doc comments on the
SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and
remove the four structural SQL comments from the alter_preserve_row_sequence
case. No behavior changes; .result regenerated by the sqlness runner.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): fail closed exact reads on copied files and extension ranges
Address review feedback on #8865:
- copy_region_from: clear the source-domain FileMeta::sequence along with
the preserve_row_sequence marker. An unmarked file retaining a stale
source-domain max sequence could be silently skipped by
files_allow_exact_sequence_range() as 'proven disjoint' in the target's
independent sequence domain, dropping rows on exact (C, H] reads. With
sequence=None the capability check fails closed (SequenceRangeUnsupported)
until the copied rows are provably disjoint.
- Engine/reader: reject exact sequence-range reads whenever a follower
region has an extension range provider attached. Extension streams are
returned without a row-level sequence filter, so exactness cannot be
proven; treat the capability as missing (fail closed) instead of emitting
out-of-range rows. The reader also fails closed as defense in depth.
Tests: extend copy_region_from regression to assert the copied file's
sequence hint is cleared; mito2 suite 1148/1148 passing.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style(mito2): use doc comments for test function descriptions
Elevate the block comments describing test functions (in scan_test and
copy_region_from_test) to /// doc comments, matching the convention used
elsewhere in the exact sequence range change. No logic change.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): extract helpers and trim comment noise in exact sequence reads
PR finalization for #8865 (zero behavior change, full mito2 suite green):
- engine: extract validate_sequence_fences and
sequence_range_unsupported_reason, keeping error variants, check order
and reason strings identical; OSS binds the extension blocker to false.
- handle_copy_region: extract remap_copied_file_meta and
file_descriptors_for_meta; rename file_ids -> source_file_ids and
files_to_copy -> new_file_metas.
- compactor: rename max_input_sequence -> known_max_input_sequence,
document the None semantics (empty input vs unknown sequence).
- Remove restating/outdated comments (ScanInput::sequence_range doc
first line, outdated file-pruning note, options test restatements),
compress verbatim comments while keeping why/invariants/contracts.
Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt,
git diff --check, mito2 suite 1148/1148 passing.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): reject foreign-region SSTs in exact sequence reads
Reading an SST whose FileMeta.region_id differs from the scanned region
means the region's sequence domain is broken (manifest corruption or a
repartition/copy path that leaked a source-domain file). Treat this as
an explicit RegionSequenceDomainBroken error instead of silently
ignoring the file's sequence or falling back to a full scan: the region
is unusable for exact sequence-range reads until the foreign lineage is
compacted away or repaired.
- files_allow_exact_sequence_range / exact_sequence_range now return
Result and propagate the error through engine fence validation and
scan construction (StatusCode::Internal, distinct from the
fallback-capable SequenceRangeUnsupported).
- Row-level flat-batch sequence filtering rejects foreign-region files
as defense in depth.
- Engine test asserts the broken-domain error rather than
Unsupported/fallback.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping
An unmarked file's FileMeta.sequence may be synthesized by the
region-edit or repartition paths (committed+1 import barrier), not a
physical max of its rows. Treating it as a whole-file disjoint proof
could permanently skip rows that were never incrementally consumed
once the flow checkpoint passes that value.
Exact sequence-range capability now requires every SST in the region to
carry the preserve_row_sequence marker; any unmarked file disables
exactness (fallback), and the (C, H] file-selection skip also only
applies to marked files. Foreign-region files still raise
RegionSequenceDomainBroken as before.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): scope exact sequence-range capability to the time-selected read set
The exact capability check used to walk the entire SstVersion, so a
single unmarked or foreign-region SST anywhere in the region disabled
exact reads or raised RegionSequenceDomainBroken even when the
request's time range could never touch that file.
Both the engine fence and the scan builder now derive the read set with
shared time-pruning + exact-min/sst-min selection and validate
capability only over the files actually selected: a time-pruned file
cannot contribute a row to (C, H], so it cannot affect exactness. The
existing fail-loud semantics are unchanged for every selected file
(foreign region id -> RegionSequenceDomainBroken; unmarked -> exact
unavailable).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): wash untrusted sequences in compaction and restore barrier skipping
Compaction with any non-preserved input now writes a sequence-less
output: the physical __sequence column is zeroed (the flat format
requires the internal columns) and FileMeta.sequence records the
region-local admission barrier committed_sequence + 1 (falling back to
the flushed frontier). preserve_row_sequence stays false.
Exact sequence-range scans interpret an unmarked file's sequence as an
admission barrier: barrier <= C means flow has already consumed the
whole file, so it is skipped at file level; a missing or newer barrier
fails closed. Foreign-region files stay in the selected read set so the
capability fence still raises RegionSequenceDomainBroken.
This closes the recovery loop: after a region repartition, one
time-scoped fallback consumes the migrated rows, then compaction washes
the untrusted per-row sequences away and exact incremental reads resume
via file-level barrier skipping.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): drop restating comments in known_max_input_sequence tests
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): trim SQLness result EOF whitespace
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): reuse exact scan file selection
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): strengthen sequence scan coverage
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* style(mito2): trim ALTER option comments
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): remove no-op bulk compaction check
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): preserve trusted row sequences when reading SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(mito2): preserve target sequence domain for imported SSTs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): add trailing blank line to SQLness result EOF
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* chore(mito2): trim exact sequence scan plumbing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(mito2): fold exact SST selection checks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): make legacy compaction rewrite deterministic
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(mito2): make PK compaction rewrite deterministic
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* ci: automate compatibility version window
Signed-off-by: discord9 <discord9@163.com>
* ci: address compat version window review feedback
Address all four review comments on the compat version window
automation:
- Keep the PR window to the sliding window only (latest patch of the two
newest stable minor lines). Exact =vX.Y.Z anchors from case.toml are no
longer unioned into from_versions; they are validated by the new
--check-anchors mode and exercised by nightly runs via --nightly-window.
- Add --published-only: the window is computed over stable git tags that
have a published, non-draft GitHub release carrying the sqlness compat
artifacts (greptime-linux-amd64 tar.gz and sha256sum), so failed releases
cannot land in the window.
- Run the updater Python tests plus the window/anchor consistency check in
PR and merge-group CI (new compat-updater-check job in integration.yml).
- Regenerate tests/compatibility/ci.toml to the current sliding window
[v1.0.2, v1.1.4].
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Lei, HUANG
LFCandgreptimedb-ci
discord9
jeremyhi
Weny Xu
