mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-06 19:06:58 +00:00
Lei, HUANG
47bf7de138
* feat(mito2): add lossless multi-region compaction executor Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): use bounded workers for multi-region compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): materialize compaction routing fragments on demand Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(index): clean up Tantivy resources on cancellation Keep temporary directories alive through initialization and in-flight writer finalization. Release writers and remove directories on the shared blocking runtime, and retain the finishing task so abort can join it safely. Cover creator lifecycle cleanup and multi-region cancellation and sibling failure after Tantivy initialization. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): stream single-window multi-region compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): read published multi-region outputs with real metadata Multi-region output metadata was only inspected field by field; every successful read cleared the sequence bound and forced the preserve marker, so partition filtering and sequence override were never exercised with the metadata a caller publishes. Read each output again with the returned FileMeta and the target metadata, and require the same row count and the same reader-visible sequences. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover multi-region routing mask rejection The exclusive-routing guards were unreachable: every fixture used a partition rule whose masks happened to be disjoint and correctly sized. An unchecked caller-supplied rule can still violate both invariants, and a violation would route one physical row to several target regions. Cover both guards: overlapping predicates from a real rule with check_exprs=false, and an over-short mask from a minimal rule implementation. Both must fail before any output is protected. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover JSON2 rewrite through multi-region compaction The multi-region executor wires the shared JSON2 rewrite plan by column id and target layout, but no fixture carried a JSON2 column, so neither the per-target mapping nor the materialized values were exercised. Write a source SST with a narrower JSON2 layout (one key stays in the remainder), compact it into a wider target layout, and read the output back: require the target physical layout and the original key values. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover every validate_plan rejection Plan validation is the boundary that keeps malformed inputs away from the routing and writer stages, but only the happy path was exercised. Mutate a valid plan for each rejection (budgets, control state, empty targets, window alignment/overflow, source ownership, duplicates, missing inputs, mismatched schemas) and require the specific reason under a timeout, with no protected outputs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): assert multi-region output bytes and retained groups output_bytes had no positive assertion, and the peak check was an upper bound that also holds when the peak is never recorded. Require the reported bytes to equal the sum of the produced files and the peak equal-key group storage to be non-zero. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(index): cover retained Tantivy finalization on abort and drop Cancellation was only exercised after finish_writer returned, so the retained finalization handle and the directory lease it holds were never tested. Simulate a commit that is still running, then require abort to join it before removing the directory and creator drop to clean up after the lease is released. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): share sequence provenance helpers on FileMeta The multi-region writer duplicated the trust predicate and maximum-bound fold that read paths and DefaultSstMerger already rely on, so the two could disagree about whether a file's row sequences are trustworthy. Move both rules onto FileMeta, delegate from FileHandle, and reuse them from the multi-region writer and the compactor. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document the guarded SST write path write_sst_with_guard is the enterprise seam whose behaviour is not visible from its signature, and the comment above the non-cache arm still claimed the cache was unconditionally disabled. Describe the bypass, the single upload stream and the pre-create guard, and correct the stale comment. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document multi-region plan budget fields The public plan left max_output_file_size and window_seconds without units or semantics, and memory_limit without the multipart-minimum floor implied by admission. State all three so callers cannot budget against the wrong unit. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): explain the multi-region row-group cap Capping output row groups at 1024 rows diverged from ordinary compaction without saying why. Record that the cap bounds the encoder working set that each resident target admits, and keep the behaviour unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): separate new error variants consistently Every other variant in the enum is separated by a blank line; the new enterprise variants were glued to their neighbours. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): quote the new module identifier in the module map Every other row of the module map starts with a backticked identifier; the multi-region row did not. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): prefix multi-region tests with test_ Every other test under compaction/ is named test_*, so name-filtered runs and IDE grouping silently skipped this module. Rename only the test functions; assertions are unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): wrap the multi-region scan stream bodies rustfmt does not reflow async_stream macro bodies, so the shared scan kept statements far past the column limit. Split them into one statement per line; behaviour is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): make the table-stage folding cases independent Both parametrized cases used identical options and differed only by merge_mode, which the table stage never reads, so the second case re-ran the first. Keep the ignored-merge-mode case and spend the second on preserve_row_sequence, asserting the per-file marker as well as the row count. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): fix partition mask test name Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): wrap multi-region worker macro bodies Wrap receive and accounting expressions and expand the scan completion arm without changing worker behavior. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document multi-region output levels Document that returned files default to L1 and callers must adjust metadata before publication when another level is intended. Distinguish this convention from changing existing persisted levels. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): identify invalid multi-region plan fields Report each invalid source or execution limit separately while preserving validation order and thresholds. Update rejection cases and cover negative window widths. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): consolidate multi-region compaction regressions Share SST fixtures and complete-row oracles while preserving non-append, JSON2, sequence, and cancellation coverage. Drive Tantivy finalization tests through the real finish path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
61 lines
2.3 KiB
TOML
61 lines
2.3 KiB
TOML
# Copyright 2023 Greptime Team
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
headerPath = "Apache-2.0.txt"
|
|
|
|
includes = [
|
|
"*.rs",
|
|
"*.py",
|
|
"*.ts",
|
|
]
|
|
|
|
excludes = [
|
|
# copied sources
|
|
"src/common/base/src/readable_size.rs",
|
|
"src/common/base/src/secrets.rs",
|
|
"src/servers/src/repeated_field.rs",
|
|
"src/servers/src/http/test_helpers.rs",
|
|
# enterprise: Apache-2.0 header is NOT applied to these. They are governed by
|
|
# the GreptimeDB Enterprise License and checked via `licenserc-enterprise.toml`.
|
|
# Keep this list in sync with that file's `includes`; enforced by
|
|
# `make check-enterprise-license`.
|
|
# enterprise:start
|
|
"src/common/meta/src/rpc/ddl/trigger.rs",
|
|
"src/operator/src/expr_helper/trigger.rs",
|
|
"src/sql/src/statements/alter/trigger.rs",
|
|
"src/sql/src/statements/create/trigger.rs",
|
|
"src/sql/src/statements/show/trigger.rs",
|
|
"src/sql/src/statements/drop/trigger.rs",
|
|
"src/sql/src/parsers/alter_parser/trigger.rs",
|
|
"src/sql/src/parsers/create_parser/trigger.rs",
|
|
"src/sql/src/parsers/show_parser/trigger.rs",
|
|
"src/mito2/src/extension.rs",
|
|
"src/mito2/src/compaction/multi_region.rs",
|
|
"src/mito2/src/compaction/multi_region/source.rs",
|
|
"src/mito2/src/compaction/multi_region/memory.rs",
|
|
"src/mito2/src/compaction/multi_region/worker.rs",
|
|
"src/mito2/src/compaction/multi_region/writer.rs",
|
|
"src/mito2/src/compaction/multi_region/tests.rs",
|
|
"src/common/function/src/admin/purge_table.rs",
|
|
"src/catalog/src/system_schema/information_schema/recycle_bin.rs",
|
|
"src/common/meta/src/ddl/purge_dropped_table.rs",
|
|
"src/common/meta/src/ddl/undrop_table.rs",
|
|
"src/catalog/src/system_schema/information_schema/recycle_bin_test.rs",
|
|
# enterprise:end
|
|
]
|
|
|
|
[properties]
|
|
inceptionYear = 2023
|
|
copyrightOwner = "Greptime Team"
|