mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-30 15:36:59 +00:00
1e97a0e7c3e9c2e6023c12bbfe4b04e283f0aaa5
49
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 fys
|
f35057d9c2 |
feat(json2): support inverted index (#9324)
feat: build inverted indexes for JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> |
||
 Lei, HUANG
|
52554db52f |
perf(mito2): keep short series unsplit in compaction merges (#9706)
* perf(mito2): keep short series unsplit in compaction merges Compaction writes every merged batch to the output SST. Splitting short series before the merge made it emit one small batch per series and input file, which the SST writer and indexer then handled one by one. Compaction now splits only when series are long enough to fill most of a batch. With 16 L0 files of 10k series x 30 rows, one output went from 160,560 merged batches to 592, and merging plus writing took 1.66s instead of 2.67s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark compacting L0 files into one output The benchmark writes 16 L0 files through the engine, each holding every series over its own time span as consecutive flushes do, and times a strict window compaction that merges them into one output. Every iteration compacts a fresh copy of the same files. With 10k series of 30 rows per file, keeping short series unsplit in compaction merges cut the time from 830ms to 520ms. With 60 series of 5,000 rows per file, which compaction still splits, both took 266ms. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 Ning Sun
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
Lei, HUANG
|
e91faa9df8 |
perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
e9dd79d1d2 |
fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes. Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): report invalid primary key ranges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): scope primary key ranges to comparisons Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts. Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle. Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): cache aligned primary key ranges per file Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes. Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify primary key mapper schema snapshot Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): align primary key range fixtures with table contract Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario. Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): enable filesystem GC in release unit tests Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito-codec): skip release value decoding in PK prefix counts Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build. Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito-codec): remove redundant prefix integration tests Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: wait for MySQL to accept authenticated TCP queries Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed. Verify readiness with SELECT 1 over TCP rather than the initialization socket. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
cb6c7a5c6d |
perf(mito-codec): streamline sparse label extraction and buffer sizing (#9217)
* perf(mito-codec): validate sparse labels once Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito-codec): reserve complete sparse label chunks Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
ab21f684bf |
perf(mito2): lazily materialize sparse primary key tags (#9199)
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations The new revision (v0y4g3r/memcomparable main, PR #1) includes: - Deserializer::read_bytes_into for reusable-buffer decoding - read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain) - bulk put_slice in both normal and reverse modes - Error::Eof instead of panic on truncated input Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): extract sparse tag values lazily without full PK decode Replace the full primary-key decode in two hot consumers of SortField::deserialize with column-lazy extraction over SparsePrimaryKeyView, eliminating the per-label owned values (Vec<Value>, one heap allocation per string) for every distinct key: - flat_format: DecodedPrimaryKeys keeps sparse keys encoded and extracts each tag column directly from the raw dictionary bytes into a reusable buffer. FlatConvertFormat (flush/compaction write path) builds all projected tag columns in one pass per key, sharing offset discovery between columns; the read path (file_range filters) keeps per-column lazy extraction. - series_index writer: builds rows from the reserved prefix and only the indexed tags via a shared offsets cache and buffer, instead of decoding all labels and copying out a few strings. - mito-codec: add SparsePrimaryKeyView::reserved_value for typed table_id/tsid extraction. Persisted formats are unchanged; values pushed to column builders are identical (NULL vs empty string, reserved column types, duplicate dictionary entries and consecutive-run order are preserved). Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion 30 samples): old = full decode + per-column builds, new = lazy extraction (+ one-pass for multi-column projection). | Workload | Before | After | Change | | --- | ---: | ---: | ---: | | 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% | | 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% | | 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% | | 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% | | 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% | Also adds a decode_sparse benchmark group to mito-codec's bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): share sparse tag scans across filters Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics. Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches. Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark sparse tag predicate count scaling Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
259b42c039 |
fix(mito2): preserve last-non-null values across compaction (#9169)
* fix(mito2): preserve last-non-null values during compaction Use a dedicated LastNonNullPicker to close TWCS seeds over overlapping SSTs, merge connected seed groups, and defer closures with busy inputs. Keep LastRow and strict-window selection unchanged. Index overlap candidates on the blocking compaction runtime and cover field promotion, transitive dependencies, expiration, and conservative key-range pruning with regression tests. Refs #9146 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard last-non-null compaction against pending memtables Track conservative memtable sequence lower bounds and capture them from the same version as compaction inputs. Defer unsafe SST closures and strict-window plans until pending writes no longer intersect their sequence span. Cache minimum sequences during bulk conversion and expose a lightweight memtable getter to avoid rescanning rows or computing full statistics. Fixes #9157 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): limit eligible last-non-null compaction outputs Enumerate all TWCS seeds for LastNonNull while retaining bounded window planning concurrency. Apply the output budget after closure expansion and busy/memtable safety checks, preserving execution priority and ordinary TWCS behavior. Cover rejected and absorbed seeds, priority ordering, and bounded and unlimited output counts with real-picker regression tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): include all candidates before last-non-null closure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify last-non-null compaction closure Extract transitive closure expansion from output scheduling and clarify seed handling. Document the dedicated picker rationale and schema-evolution risks in PK overlap checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover deferred non-seed compaction bridges Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): defer strict-window compaction with busy dependencies Keep busy SSTs when building the LastNonNull strict-window dependency closure, then defer the entire rewrite plan if any selected input is busy. Preserve LastRow filtering and expired-file cleanup.\n\nCover requested, transitive and unrelated busy windows, including retry after the dependency is released. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document compaction overlap index fields Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
d6a8974613 |
perf(mito2): optimize materialized column index updates (#9182)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
7aacc947b3 |
perf(mito2): lazily extract sparse primary key index values (#9176)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
c3ea022de5 |
perf(mito2): blazing-fast tournament tree merger (#8989)
* perf(mito2): optimize flat merge heap and primary-key interleave Replace the per-row BinaryHeap pop/push cycle in FlatMerge with an in-place root mutation plus a single sift-down repair on a custom RootHeap, keeping the cold heap and direct-batch fast path unchanged. Fallible or awaiting batch transitions move the hot node out of the heap first, preserving error and cancellation semantics. Exploit the globally sorted merge output to build the internal Dictionary<UInt32, Binary> primary-key column with a one-pass ordered gather: append a Binary value only when the PK changes and reuse the current key for adjacent equal PKs, bypassing Arrow dictionary masks, hash interning and key remapping. Non-PK columns still use Arrow interleave. Also cache the current primary-key byte range in RowCursor to avoid repeated dictionary range decoding during comparisons, and add a setup-free Criterion benchmark with exact output-row assertions. 32-way/1 row-per-series/40-tag improves 955.79ms -> 562.36ms (-41.2%); 0-tag -39.5%, 64 rows/series -79.8%, 8-way -56.1%, single-iterator control +0.3%. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add rows-per-series sweep to flat merge bench Add 32-way/40-tag shapes for 1, 10, 100, 1000 and 10000 rows per series, and allow FLAT_MERGE_BENCH_SHAPE to match shape name prefixes so the whole sweep can run in one invocation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): add oracle-based correctness tests for RootHeap Drive RootHeap and a std BinaryHeap oracle with the same seeded op sequence (push / pop / mutate-root + repair) and assert peek, len, best_child and the full drain order after every operation. A second run with a tiny value range makes duplicates dominate, covering the equal-key branches of sift_up/sift_down and best_child. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): replace hot heap with a tournament tree in flat merge Replace the hot RootHeap with a fixed-capacity tournament (winner) tree over per-node slots: every internal node caches the champion of its subtree, so advancing the winner only replays the ~log2(k) nodes on its leaf-to-root path with one compare per level, instead of the heap's two-compares-per-level sift that also re-compares the same node pairs on every row. Two fast paths keep dense shapes at O(1) per row: - champion retention: after mutating the winner in place, skip the replay entirely when it still beats the runner-up (its path caches are unchanged by construction); - a second-best slot cache, invalidated on any structural change, so the retention check costs a single compare without walking the tree. The cold heap, hot/cold overlap window, direct-batch fast path and the remove-before-fallible-fetch batch transition semantics are unchanged. Vs the RootHeap version: 1rps/32way/40tag -19.7%, 0tag -34.4%, 8way -15.9%, 64rps -30.5%, sweep 10/100/1000/10000rps -29~32%; vs the original BinaryHeap baseline the main shape is -52.8%. The single-iterator control is +8% (+50ns one-time construction allocation, no merge work). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): support generic schemas in flat merge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): satisfy clippy in flat merge benchmark Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): cache flat merge primary key index Compute the internal primary-key column index once when constructing BatchBuilder and reuse it for every output batch. Preserve the column-name gate for generic schemas. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark high-fan-in flat merges Add sparse 64, 128, 256, and 512-way merge shapes while keeping the total input fixed at 3.2 million rows. Compared with the merge-base heap implementation, median time improves by 56.0%, 60.8%, 55.6%, and 56.1%, respectively. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
9cfbc42126 |
refactor(mito2): prioritize file count in TWCS picker (#8765)
* refactor(mito2): prioritize file count in TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): migrate TWCS tests to async picker API Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove legacy reduce_runs and merge_seq_files pickers Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): balance TWCS picks by file group Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): treat multi-SST groups as barriers in TWCS pick_count_first Previously pick_count_first filtered out multi-SST file groups and could pick singleton groups across them in one interval. Now multi-SST groups split the candidates into independent segments, so a single pick never crosses a multi-SST group. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant overlaps_files_left_behind check The rebase onto main kept both the upstream fix (#8872) and the branch's selected_overlaps_unselected check for the same deletion-marker problem. The upstream check is fully subsumed: files_to_merge only differs from the window files in append mode, where filter_deleted is already false, and selected_overlaps_unselected treats partially-selected groups as unselected, making it strictly stronger. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): pre-filter by selection time span in selected_overlaps_unselected A window group outside the overall time span of the selected groups cannot overlap any of them, so skip the precise overlap check (and the per-group file id set lookup) for it. In typical TWCS windows the pick is clustered in one segment, so most unselected groups are rejected in O(1). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove FileGroup abstraction from TWCS picker Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): drain compactable backlog after successful compaction Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): score TWCS candidates by predicted progress Redefine what a pick candidate is worth: an interval is only eligible if compacting it makes progress on at least one axis — it reduces the physical file count given the max_output_file_size split threshold (predicted output = ceil(input bytes / threshold)), or it resolves at least one overlap between sorted runs. A pure rewrite that achieves neither (e.g. 32 large balanced files whose output would split back into just as many SSTs) is skipped instead of burning I/O. The candidate metrics are accumulated in a Candidate struct as the interval expands, and the score ranks by predicted file reduction, then overlap participants, then smaller input bytes. With no output size limit the behavior degenerates to the previous count-first rule. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): make TWCS input limit configurable Allow operators to tune the maximum SST inputs through a hidden environment variable while keeping 32 as the validated default. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): apply TWCS trigger at window level Check the total physical SST count before candidate selection so trigger values above the per-task input limit still allow bounded compactions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 Lanqing Yang
|
0e28916695 |
perf(mito2): optimize dictionary primary key sorting (#8767)
Signed-off-by: lyang24 <lanqingy93@gmail.com> |
||
 Yingwen
|
5b1b12d041 |
fix(mito2): adapt batch size for wide rows (#8543)
* fix(mito2): adapt batch size for wide rows Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comments Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): simplify bulk memtable merging Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix batch size license header Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): size encoded batches from row groups Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lanqing Yang
|
565899e482 |
perf(mito): cached-size single-pass WAL entry encoder (#8254)
prost has no size caching, so WalEntry::encode_to_vec recomputes the encoded_len of every nested message for each length delimiter. In the deep WAL tree (WalEntry/Mutation/Rows/Row/Value) a leaf Value's length is recomputed once per ancestor level (~5x). Add WalEntryEncoder: one size pass caches every message body length into a flat vector (pre-order), one encode pass writes bytes reading the cached lengths back via a cursor, so each length is computed exactly once. Leaf messages (Value, ColumnSchema, WriteHint, BulkWalEntry) delegate their body encoding to prost but have their single computed length cached. Output is byte-for-byte identical to encode_to_vec (asserted in tests covering delete op_type, null values, empty entry, encoder reuse), which is required for WAL replay compatibility. Wired into WalWriter::add_entry, reused per write batch. This compensates for prost lacking a cached_size mechanism and can be removed if such caching lands upstream. Signed-off-by: lyang24 <lanqingy93@gmail.com> |
||
  LFCandcopilot-swe-agent[bot]
|
59b738d7f2 |
feat: compact json2 data (#8103)
Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Lei, HUANG
|
0a2400cbb9 |
fix: optimize sorted run picking (#8128)
* fix/optimize-pick: Optimize sorted run picking - `find_sorted_runs`: prune inactive run candidates and keep `find_sorted_runs_original` for testing in `src/mito2/src/compaction/run.rs` - `bench_compaction_picker`: compare old and new sorted-run picking with PK-aware workloads in `src/mito2/benches/bench_compaction_picker.rs` - `features`: expose `testing` for benchmark-only helpers in `src/mito2/Cargo.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix/optimize-pick: Remove redundant overlap guard - `find_sorted_runs`: simplify active candidate overlap checks in `src/mito2/src/compaction/run.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: apply review comment Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
7840aa1bb4 |
refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk Signed-off-by: evenyag <realevenyag@gmail.com> * chore: keep partition tree memtable for migration test Restore PartitionTreeMemtable construction when memtable.type=partition_tree is explicit, and move the sparse-encoding bulk override into the default (no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse, so the test actually exercises a PartitionTreeMemtable in phase 1 and verifies WAL replay into the new BulkMemtable on reopen without flushing. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: drop partition tree memtable from runtime Re-apply the unconditional sparse-encoding override in `MemtableBuilderProvider::builder_for_options` and route the `MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation warning. After this change, `PartitionTreeMemtableBuilder` is no longer reachable from the engine runtime; benchmarks still reference the type. Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the `put_sparse_rows` helper added in the previous commit — that test only existed to validate the PartitionTree -> Bulk reopen migration and is unnecessary now that the override is in place. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): move timestamp_array_to_i64_slice into read module Relocate the timestamp_array_to_i64_slice helper from memtable/partition_tree/data.rs to the read module so that the read path no longer depends on the partition_tree internals. All call sites (both inside and outside the partition_tree module) now import from crate::read. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests The time_partition tests use the memtable builder purely as a generic backend for the TimePartitions write/scan paths; nothing in them is specific to the partition-tree memtable. Switch the seven affected tests to TimeSeriesMemtableBuilder so the tests no longer depend on PartitionTreeMemtableBuilder. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): delete PartitionTreeMemtable implementation The runtime already falls back to BulkMemtable for the PartitionTree variant. Drop the now-unreachable implementation, its metrics, the partition_tree benchmarks, the metric-engine Unsupported fallback in bulk_insert.rs, and the test helpers that only existed for the deleted module. MemtableOptions::PartitionTree, its parsing, the runtime fallback, the store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures remain so existing region options keep round-tripping. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito-codec): drop skip_partition_column parameter PartitionTreeMemtable was the only caller passing skip_partition_column=true; every other caller passes false. Now that the partition_tree module is gone, the parameter is uniformly false and the guard branch is dead. Drop the parameter from the trait method and both impls, remove the guard and the is_partition_column helper, and update the four remaining call sites in mito2 plus the bench. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove unused MemtableConfig enum Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused variant Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test_config_api Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove unused memtable test helpers Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: support bulk memtable options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove partition tree options from region options Move primary_key_encoding to the top level Signed-off-by: evenyag <realevenyag@gmail.com> * test: make ssts test datetime replaced text stable Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * chore: validate_enum_options consider bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass region id when parsing region options Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with `try_from_options(region_id, options_map)` so the legacy partition_tree fallback can log the affected region. The fallback now also overrides the SST format to flat in addition to clearing the memtable type. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: align sst_format with bulk memtable on parse and open Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
fd94f55193 |
refactor(mito2): remove dead scan code (#7925)
* refactor(mito2): remove dead batch parallel scan helpers Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): remove dead merge reader path Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): remove dead batch dedup reader Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): remove obsolete batch source helper Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused plain batch Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
0e22d6a72b |
feat: implement partition range cache stream (#7842)
* feat: add cache stream helpers, key construction, config wiring, and metrics for partition range cache Add range result cache size config field and wire it through cache builder chains. Implement cache key building (build_range_cache_key), stream replay/store helpers (cached_flat_range_stream, cache_flat_range_stream), dictionary compaction (compact_pk_dictionary), and partition range row group collection. Add range cache metrics (size, hit, miss) to ScanMetricsSet and PartitionMetrics. Move fingerprint tests from scan_region to range_cache module. These functions are not yet wired into scan execution. Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add benchmark for cache stream Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: move bench_util to test_util Signed-off-by: evenyag <realevenyag@gmail.com> * feat: share dict Signed-off-by: evenyag <realevenyag@gmail.com> * test: test ptr_eq Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify value array handling Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add todo for estimate size Signed-off-by: evenyag <realevenyag@gmail.com> * feat: simplify size calculation Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove one test Signed-off-by: evenyag <realevenyag@gmail.com> * test: update config test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Only ignore exprs that can extract time ranges Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix tests Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
be4a7a6d37 |
refactor: remove Memtable::iter (#7809)
* refactor: remove Memtable::iter Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: review comments Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
04cd2c8a05 |
feat: flat read path support primary_key format memtables (#7759)
* feat: add adapter for batch to flat recordbatch Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support batch to flat record batch in MemtableRange Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: address review issues for BatchToRecordBatchAdapter - Extract duplicated read_column_ids computation into a shared `read_column_ids_from_projection` helper function - Cache `FormatProjection` in `BatchToRecordBatchContext::new()` instead of recomputing it on every `adapt_iter()` call - Remove unnecessary `Arc` wrapping of `read_column_ids` in `SimpleBulkMemtable::ranges()` - Fix clippy `filter_map_bool_then` warning in `batch_adapter.rs` Signed-off-by: evenyag <realevenyag@gmail.com> * chore: simplify comments Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use read column ids in batch adapter Signed-off-by: evenyag <realevenyag@gmail.com> * test: test build_record_batch_iter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * test: test build_record_batch_iter for all old memtables Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: prune time range before adapter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: share BatchToRecordBatchContext in simple_bulk_memtable.rs Signed-off-by: evenyag <realevenyag@gmail.com> * chore: use ScalarValue::to_array_of_size to build repeated value array Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
LFC
|
5eac4f10aa |
chore: remove dependency on "atty" (#7725)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
7711661618 |
feat: BulkMemtable compact parts without encoding into Parquet (#7617)
* feat: implement MultiBulkPart to hold a list of batches in BulkMemtable Signed-off-by: evenyag <realevenyag@gmail.com> * feat: Only encode parts when there are enough rows Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: merge MultiBulkPartIter and BulkPartBatchIter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove some enums and structs Signed-off-by: evenyag <realevenyag@gmail.com> * chore: reuse code in merging bulk/encoded parts Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: collect part groups directly Signed-off-by: evenyag <realevenyag@gmail.com> * test: add unit tests Signed-off-by: evenyag <realevenyag@gmail.com> * feat: enlarge merge threshold and configure by env - GREPTIME_BULK_MERGE_THRESHOLD - GREPTIME_BULK_ENCODE_ROW_THRESHOLD - GREPTIME_BULK_ENCODE_BYTES_THRESHOLD Signed-off-by: evenyag <realevenyag@gmail.com> * feat: change flush strategy Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add BulkMemtableConfig Signed-off-by: evenyag <realevenyag@gmail.com> * chore: limit max groups and adjust threshold Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add flush file number metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add bulk filter 1 host bench Signed-off-by: evenyag <realevenyag@gmail.com> * feat: adjust bulk compact threshold Signed-off-by: evenyag <realevenyag@gmail.com> * feat: flush a file if == min_flush_rows Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix test_index_build_type_compact test Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix mito tests Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove regions from catchup_regions before notify Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
a22d08f1b1 |
feat: collect merge and dedup metrics (#7375)
* feat: collect FlatMergeReader metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add MergeMetricsReporter, rename Metrics to MergeMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove num_input_rows from MergeMetrics The merge reader won't dedup so there is no need to collect input rows Signed-off-by: evenyag <realevenyag@gmail.com> * feat: report merge metrics to PartitionMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add dedup cost to DedupMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect dedup metrics Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove metrics from FlatMergeIterator Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove num_output_rows from MergeMetrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement merge() for merge and dedup metrics Signed-off-by: evenyag <realevenyag@gmail.com> * fix: report metrics after observe metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
3001c2d719 |
feat: BulkMemtable stores small fragments in another buffer (#7164)
* feat: buffer small parts in bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use assert_eq instead of assert Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * chore: collect bulk memtable scan metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: report metrics early Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
acf38a7091 |
fix: avoid filtering rows with delete op by fields under merge mode (#7154)
* chore: clear allow dead_code for flat format
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: pass exprs to build appliers
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: split field filters and index appliers
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support skip filtering fields in RowGroupPruningStats
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: add PreFilterMode to config whether to skip filtering fields
Adds the PreFilterMode to the RangeBase and sets it in
ParquetReaderBuilder
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support skipping fields in prune reader
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support pre filter mode in bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: pass PreFilterMode to memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: test mito filter delete
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fix compiler errors
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: remove commented code
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: move predicate and sequence to RangesOptions
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fmt code
Signed-off-by: evenyag <realevenyag@gmail.com>
* ci: skip cargo gc
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fix cargo build warning
Signed-off-by: evenyag <realevenyag@gmail.com>
* Revert "ci: skip cargo gc"
This reverts commit
|
||
|
Yingwen
|
cff9cb6327 |
feat: converts batches in old format to the flat format in query time (#6987)
* feat: use correct projection index for old format Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove allow dead_code from format Signed-off-by: evenyag <realevenyag@gmail.com> * feat: check and convert old format to flat format Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sub primary key num from projection Signed-off-by: evenyag <realevenyag@gmail.com> * fix: always convert the batch in FlatRowGroupReader Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: Change &Option<&[]> to Option<&[]> Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: only build arrow schema once adds a method flat_sst_arrow_schema_column_num() to get the field num Signed-off-by: evenyag <realevenyag@gmail.com> * feat: Handle flat format and old format separately Adds two structs ParquetFlat and ParquetPrimaryKeyToFlat. ParquetPrimaryKeyToFlat delegates stats and projection to the PrimaryKeyReadFormat. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: handle non string tag correctly Signed-off-by: evenyag <realevenyag@gmail.com> * fix: do not register file cache twice Signed-off-by: evenyag <realevenyag@gmail.com> * fix: clean temp files Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add rows and bytes to flush success log Signed-off-by: evenyag <realevenyag@gmail.com> * chore: convert format in memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: add compaction flag to ScanInput Signed-off-by: evenyag <realevenyag@gmail.com> * fix: compaction should use old format for sparse encoding Signed-off-by: evenyag <realevenyag@gmail.com> * fix: merge schema use old format in sparse encoding Signed-off-by: evenyag <realevenyag@gmail.com> * feat: reads legacy format but not convert if skip_auto_convert Signed-off-by: evenyag <realevenyag@gmail.com> * fix: suppport sparse encoding in bulk parts Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
9c22092189 |
feat: Implements compaction for bulk memtable (#6923)
* feat: initial bulk memtable compaction implementation Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement compact for memtable Signed-off-by: evenyag <realevenyag@gmail.com> * test: add tests for bulk memtable compaction Signed-off-by: evenyag <realevenyag@gmail.com> * style: clippy Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comments Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Ruihang Xia
|
c9377e7c5a |
build: bump rust edition to 2024 (#6920)
* bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Yingwen
|
8fc3a9a9d7 |
feat: Implements async FlatMergeReader and FlatDedupReader (#6761)
* refactor: Add Flat prefix to MergeIterator and DedupIterator Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement MergeReader for RecordBatch Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use GenericNode for IterNode Signed-off-by: evenyag <realevenyag@gmail.com> * feat: flat merge reader to stream Signed-off-by: evenyag <realevenyag@gmail.com> * feat: implement FlatDedupReader Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add a benchmark for FlatMergeIterator Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename plain_projection to flat_projection Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
39e2f122eb |
feat: EncodedBulkPartIter iters flat format and returns RecordBatch (#6655)
* feat: implements iter to read bulk part Signed-off-by: evenyag <realevenyag@gmail.com> * feat: BulkPartEncoder encodes BulkPart instead of mutation Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
50f7f61fdc |
feat: Implements an iterator to read the RecordBatch in BulkPart (#6647)
* feat: impl RecordBatchIter for BulkPart Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename BulkPartIter to EncodedBulkPartIter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add iter benchmark Signed-off-by: evenyag <realevenyag@gmail.com> * feat: filter by primary key columns Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: move struct definitions Signed-off-by: evenyag <realevenyag@gmail.com> * feat: bulk iter for flat schema Signed-off-by: evenyag <realevenyag@gmail.com> * feat: iter filter benchmark Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * fix: use corrent sequence array to compare Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove RecordBatchIter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comments Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * feat: apply projection first Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address comment No need to check number of rows after filter Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Yingwen
|
52466fdd92 |
feat: Implement a converter to converts KeyValues into BulkPart (#6620)
* chore: add api to memtable to check bulk capability Signed-off-by: evenyag <realevenyag@gmail.com> * feat: Add a converter to convert KeyValues into BulkPart Signed-off-by: evenyag <realevenyag@gmail.com> * feat: move supports_bulk_insert to MemtableBuilder Signed-off-by: evenyag <realevenyag@gmail.com> * chore: benchmark Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use write_bulk if the memtable benefits from it Signed-off-by: evenyag <realevenyag@gmail.com> * test: test BulkPartConverter Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add a flag to store unencoded primary keys Signed-off-by: evenyag <realevenyag@gmail.com> * feat: cache schema for converter Implements to_flat_sst_arrow_schema Signed-off-by: evenyag <realevenyag@gmail.com> * chore: simplify tests Signed-off-by: evenyag <realevenyag@gmail.com> * fix: don't use bulk convert branch now Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comments * simplify primary_key_column_builders check * return error if value is not string Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add FlatSchemaOptions::from_encoding and test sparse encoding Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
c3201c32c3 |
feat(mito): replace Memtable::iter with Memtable::ranges (#6549)
* bulk-multiparts-merge-reader: **Enhance Memtable Iteration and Flushing Logic** - **`flush.rs`**: Updated `RegionFlushTask` to handle multiple ranges using `MergeReaderBuilder` for improved source management during flush operations. - **`memtable.rs`**: Introduced `build_prune_iter` and `build_iter` methods in `MemtableRange` for flexible iteration. Added `MemtableRanges` struct to manage multiple contexts. - **`simple_bulk_memtable.rs`**: Refactored to use `BatchIterBuilder` and `BatchIterBuilderDeprecated` for iteration, supporting new `read_to_values` method in `Series`. - **`time_series.rs`**: Added `read_to_values` and `finish_cloned` methods in `Series` and `ValueBuilder` for efficient data handling. - **`scan_util.rs`**: Replaced `build_iter` with `build_prune_iter` for range iteration, enhancing scan utility. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: - **Add Rayon for Parallel Processing**: Introduced `rayon` for parallel processing in `simple_bulk_memtable.rs` and updated `Cargo.toml` and `Cargo.lock` to include `rayon` dependency. - **Enhance Benchmarking**: Added new benchmarks in `simple_bulk_memtable.rs` to compare parallel vs sequential processing, projection, sequence filtering, and write performance. - **Make Structs and Methods Public**: Changed visibility of several structs and methods to `pub` in `simple_bulk_memtable.rs`, `memtable.rs`, `time_series.rs`, and `test_util.rs` to facilitate testing and benchmarking. - **Update Criterion Features**: Modified `Cargo.toml` to include `html_reports` feature for `criterion`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: ### Commit Summary - **Refactor `SimpleBulkMemtable`**: - Moved `ranges_sequential` function to a new `test_only` module and made it a method of `SimpleBulkMemtable`. - Made several fields in `SimpleBulkMemtable` private and added a `region_metadata` getter. - Affected files: `simple_bulk_memtable.rs`, `test_only.rs`. - **Benchmark Adjustments**: - Updated benchmark functions to use the new `ranges_sequential` method. - Affected file: `simple_bulk_memtable.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: ### Add Test Configuration for `iter` Method in Memtable Implementations - **Enhancements**: - Added `#[cfg(any(test, feature = "test"))]` attribute to the `iter` method in various `Memtable` implementations to enable conditional compilation for testing purposes. - Affected files: - `src/mito2/src/memtable.rs` - `src/mito2/src/memtable/bulk.rs` - `src/mito2/src/memtable/partition_tree.rs` - `src/mito2/src/memtable/simple_bulk_memtable.rs` - `src/mito2/src/memtable/time_series.rs` - `src/mito2/src/test_util/memtable_util.rs` - **Benchmark Adjustments**: - Removed `black_box` usage in `bench_memtable_write_performance` function to streamline benchmarking. - Affected file: `src/mito2/benches/simple_bulk_memtable.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: **Enhance Async Support and Refactor Iteration in `mito2`** - **Add Async Features**: Updated `Cargo.toml` to include `async` and `async_tokio` features for `criterion`. - **Async Iteration**: Introduced async functions `flush` and `flush_original` in `simple_bulk_memtable.rs` to handle memtable flushing using async iterators. - **Refactor Iteration Logic**: Moved `create_iter` and `BatchIterBuilderDeprecated` to `test_only.rs` for better separation of concerns. - **Public API Change**: Made `next_batch` in `read.rs` public to support async batch processing. - **Benchmark Updates**: Modified benchmarks in `simple_bulk_memtable.rs` to use async runtime for performance testing. Files affected: `Cargo.toml`, `simple_bulk_memtable.rs`, `test_only.rs`, `read.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: **Enhance Benchmarking for Memtable** - Refactored `create_large_memtable` to `create_memtable_with_rows` in `simple_bulk_memtable.rs` to allow dynamic row count configuration. - Introduced parameterized benchmarking in `bench_ranges_parallel_vs_sequential` to test various row counts, improving the flexibility and coverage of performance tests. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: ### Enhance Memory Management and Public API - **`builder.rs`**: Made `next_offset` method public to allow external access to offset calculations. - **`simple_bulk_memtable.rs`**: Simplified the `series.extend` method by removing the iterator conversion for `fields`. - **`time_series.rs`**: - Added `can_accommodate` method to `ValueBuilder` to check if fields can be accommodated without offset overflow. - Modified `extend` method to use a `Vec` for `fields` instead of an iterator, improving memory management and error handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: Add License and Enhance Testing in `simple_bulk_memtable.rs` - Added Apache License header to `simple_bulk_memtable.rs`. - Modified test configuration in `simple_bulk_memtable.rs` to include `any(test, feature = "test")`. - Introduced a new test `test_write_read_large_string` in `simple_bulk_memtable.rs` to verify handling of large strings. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: Update `Cargo.toml` dependencies - Adjust features for `common-meta` and `mito-codec` to include "testing". - Maintain `criterion` version and features for async support. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: ### Update Predicate Type in Memtable Iterators - **Files Modified**: - `src/mito2/src/memtable.rs` - `src/mito2/src/memtable/bulk.rs` - `src/mito2/src/memtable/simple_bulk_memtable.rs` - **Key Changes**: - Updated the `iter` method in `Memtable` trait and its implementations to use `Option<table::predicate::Predicate>` instead of `Option<Predicate>`. - Adjusted return type in `BulkMemtable`'s `iter` method to `Result<crate::memtable::BoxedBatchIterator>`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: **Enhance Memtable Functionality** - **`memtable.rs`**: - Added `Clone` trait to `MemtableStats` and made `num_ranges` public. - Introduced `num_rows` field in `MemtableRange` and updated its constructor. - Added `num_rows` method to `MemtableRange`. - **`partition_tree.rs`, `simple_bulk_memtable.rs`, `time_series.rs`**: - Updated `MemtableRange` instantiation to include `num_rows`. - **`range.rs`**: - Refactored `MemRangeBuilder` to handle a single `MemtableRange` and `MemtableStats`. - **`scan_region.rs`**: - Enhanced memtable filtering based on time range and updated `MemRangeBuilder` usage. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: **Enhancements and Bug Fixes** - **Deduplication Enhancements**: - Introduced `DedupReader` and `LastRow` as public structs in `dedup.rs` to enhance deduplication capabilities. - Added `LastNonNull` deduplication strategy in `flush.rs` and `simple_bulk_memtable.rs`. - **Memtable Improvements**: - Updated `SimpleBulkMemtable` to support batch size configuration and deduplication strategies. - Modified `Series` struct in `time_series.rs` to include a configurable capacity. - **Testing Enhancements**: - Added new test `test_write_dedup` in `simple_bulk_memtable.rs` to verify deduplication functionality. - Updated existing tests to include `OpType` parameter for better operation type handling. - **Refactoring**: - Renamed `BatchIterBuilder` to `BatchRangeBuilder` in `simple_bulk_memtable.rs` for clarity. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * bulk-multiparts-merge-reader: - **Refactor `flush.rs`:** Removed `LastNonNullIter` usage and adjusted `DedupReader` instantiation to use `LastRow::new(false)` and `LastNonNull::new(false)`. - **Enhance `simple_bulk_memtable.rs`:** Added logic to handle `LastNonNull` merge mode in `IterBuilder`. Introduced new tests: `test_delete_only` and `test_single_range` to verify delete operations and single range handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: tests Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
eaf1e1198f |
refactor: Extract mito codec part into a new crate (#6307)
* chore: add a new crate mito-codec Signed-off-by: evenyag <realevenyag@gmail.com> * feat: port necessary mods for primary key codec Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use codec utils in mito-codec Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused mods Signed-off-by: evenyag <realevenyag@gmail.com> * style: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove Partition::is_partition_column() Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove duplicated test utils Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove unused comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix is_partition_column check Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
4e615e8906 |
feat(wal): support bulk wal entries (#6178)
* feat/bulk-wal: ### Refactor: Simplify Data Handling in LogStore Implementations - **`kafka/log_store.rs`, `raft_engine/log_store.rs`, `wal.rs`, `raw_entry_reader.rs`, `logstore.rs`:** - Refactored `entry` and `build_entry` functions to accept `Vec<u8>` directly instead of `&mut Vec<u8>`. - Removed usage of `std::mem::take` for data handling, simplifying the code and improving readability. - Updated test cases to align with the new function signatures. * feat/bulk-wal: ### Add Support for Bulk WAL Entries and Flight Data Encoding - **Add `raw_data` field to `BulkPart` and related structs**: Updated `BulkPart` and related structures in `src/mito2/src/memtable/bulk/part.rs`, `src/mito2/src/memtable/simple_bulk_memtable.rs`, `src/mito2/src/memtable/time_partition.rs`, `src/mito2/src/region_write_ctx.rs`, `src/mito2/src/worker/handle_bulk_insert.rs`, and `src/store-api/src/region_request.rs` to include a new `raw_data` field for handling Arrow IPC data. - **Implement Flight Data Encoding**: Added a new module `flight` in `src/common/test-util/src/flight.rs` to encode record batches to Flight data format. - **Update `greptime-proto` dependency**: Changed the revision of the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml`. - **Enhance WAL Writer and Tests**: Modified `src/mito2/src/wal.rs` and related test files to support bulk WAL entries and added tests for encoding and handling bulk data. * feat/bulk-wal: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`. - **Add `common-grpc` Dependency**: Added `common-grpc` as a dependency in `Cargo.lock` and `src/mito2/Cargo.toml`. - **Refactor `BulkPart` Structure**: Removed `num_rows` field and added `num_rows()` method in `src/mito2/src/memtable/bulk/part.rs`. Updated related usages in `src/mito2/src/memtable/simple_bulk_memtable.rs`, `src/mito2/src/memtable/time_partition.rs`, `src/mito2/src/memtable/time_series.rs`, `src/mito2/src/region_write_ctx.rs`, and `src/mito2/src/worker/handle_bulk_insert.rs`. - **Implement `TryFrom` and `From` for `BulkWalEntry`**: Added implementations for converting between `BulkPart` and `BulkWalEntry` in `src/mito2/src/memtable/bulk/part.rs`. - **Handle Bulk Entries in Region Opener**: Added logic to process bulk entries in `src/mito2/src/region/opener.rs`. - **Fix `BulkInsertRequest` Handling**: Corrected `region_id` handling in `src/operator/src/bulk_insert.rs` and `src/store-api/src/region_request.rs`. - **Add Error Variant for `ConvertBulkWalEntry`**: Added a new error variant in `src/mito2/src/error.rs` for handling bulk WAL entry conversion errors. * fix: ci * feat/bulk-wal: Add bulk write operation in `opener.rs` - Enhanced the region write context by adding a call to `write_bulk()` after `write_memtable()` in `opener.rs`. - This change aims to improve the efficiency of writing operations by enabling bulk writes. * feat/bulk-wal: Enhance error handling and metrics in `bulk_insert.rs` - Updated `Inserter` to improve error handling by capturing the result of `datanode.handle(request)` and incrementing the `DIST_INGEST_ROW_COUNT` metric with the number of affected rows. * feat/bulk-wal: ### Remove Encode Error Handling for WAL Entries - **`error.rs`**: Removed the `EncodeWal` error variant and its associated handling. - **`wal.rs`**: Eliminated the `entry_encode_buf` buffer and its usage for encoding WAL entries. Replaced with direct encoding to a vector using `encode_to_vec()`. |
||
|
Lei, HUANG
|
4b71e493f7 |
feat!: revise compaction picker (#6121)
* - **Refactor `RegionFilePathFactory` to `RegionFilePathProvider`:** Updated references and implementations in `access_layer.rs`, `write_cache.rs`, and related test files to use the new struct name. - **Add `max_file_size` support in compaction:** Introduced `max_file_size` option in `PickerOutput`, `SerializedPickerOutput`, and `WriteOptions` in `compactor.rs`, `picker.rs`, `twcs.rs`, and `window.rs`. - **Enhance Parquet writing logic:** Modified `parquet.rs` and `parquet/writer.rs` to support optional `max_file_size` and added a test case `test_write_multiple_files` to verify writing multiple files based on size constraints. **Refactor Parquet Writer Initialization and File Handling** - Updated `ParquetWriter` in `writer.rs` to handle `current_indexer` as an `Option`, allowing for more flexible initialization and management. - Introduced `finish_current_file` method to encapsulate logic for completing and transitioning between SST files, improving code clarity and maintainability. - Enhanced error handling and logging with `debug` statements for better traceability during file operations. - **Removed Output Size Enforcement in `twcs.rs`:** - Deleted the `enforce_max_output_size` function and related logic to simplify compaction input handling. - **Added Max File Size Option in `parquet.rs`:** - Introduced `max_file_size` in `WriteOptions` to control the maximum size of output files. - **Refactored Indexer Management in `parquet/writer.rs`:** - Changed `current_indexer` from an `Option` to a direct `Indexer` type. - Implemented `roll_to_next_file` to handle file transitions when exceeding `max_file_size`. - Simplified indexer initialization and management logic. - **Refactored SST File Handling**: - Introduced `FilePathProvider` trait and its implementations (`WriteCachePathProvider`, `RegionFilePathFactory`) to manage SST and index file paths. - Updated `AccessLayer`, `WriteCache`, and `ParquetWriter` to use `FilePathProvider` for path management. - Modified `SstWriteRequest` and `SstUploadRequest` to use path providers instead of direct paths. - Files affected: `access_layer.rs`, `write_cache.rs`, `parquet.rs`, `writer.rs`. - **Enhanced Indexer Management**: - Replaced `IndexerBuilder` with `IndexerBuilderImpl` and made it async to support dynamic indexer creation. - Updated `ParquetWriter` to handle multiple indexers and file IDs. - Files affected: `index.rs`, `parquet.rs`, `writer.rs`. - **Removed Redundant File ID Handling**: - Removed `file_id` from `SstWriteRequest` and `CompactionOutput`. - Updated related logic to dynamically generate file IDs where necessary. - Files affected: `compaction.rs`, `flush.rs`, `picker.rs`, `twcs.rs`, `window.rs`. - **Test Adjustments**: - Updated tests to align with new path and indexer management. - Introduced `FixedPathProvider` and `NoopIndexBuilder` for testing purposes. - Files affected: `sst_util.rs`, `version_util.rs`, `parquet.rs`. * chore: rebase main * feat/multiple-compaction-output: ### Add Benchmarking and Refactor Compaction Logic - **Benchmarking**: Added a new benchmark `run_bench` in `Cargo.toml` and implemented benchmarks in `benches/run_bench.rs` using Criterion for `find_sorted_runs` and `reduce_runs` functions. - **Compaction Module Enhancements**: - Made `run.rs` public and refactored the `Ranged` and `Item` traits to be public. - Simplified the logic in `find_sorted_runs` and `reduce_runs` by removing `MergeItems` and related functions. - Introduced `find_overlapping_items` for identifying overlapping items. - **Code Cleanup**: Removed redundant code and tests related to `MergeItems` in `run.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Add Benchmarks - **Compaction Logic Improvements**: - Updated `reduce_runs` function in `src/mito2/src/compaction/run.rs` to remove the target parameter and improve the logic for selecting files to merge based on minimum penalty. - Enhanced `find_overlapping_items` to handle unsorted inputs and improve overlap detection efficiency. - **Benchmark Enhancements**: - Added `bench_find_overlapping_items` in `src/mito2/benches/run_bench.rs` to benchmark the new `find_overlapping_items` function. - Extended existing benchmarks to include larger data sizes. - **Testing Enhancements**: - Updated tests in `src/mito2/src/compaction/run.rs` to reflect changes in `reduce_runs` and added new tests for `find_overlapping_items`. - **Logging and Debugging**: - Improved logging in `src/mito2/src/compaction/twcs.rs` to provide more detailed information about compaction decisions. * feat/multiple-compaction-output: ### Refactor and Enhance Compaction Logic - **Refactor `find_overlapping_items` Function**: Changed the function signature to accept slices instead of mutable vectors in `run.rs`. - **Rename and Update Struct Fields**: Renamed `penalty` to `size` in `SortedRun` struct and updated related logic in `run.rs`. - **Enhance `reduce_runs` Function**: Improved logic to sort runs by size and limit probe runs to 100 in `run.rs`. - **Add `merge_seq_files` Function**: Introduced a new function `merge_seq_files` in `run.rs` for merging sequential files. - **Modify `TwcsPicker` Logic**: Updated the compaction logic to use `merge_seq_files` when only one run is found in `twcs.rs`. - **Remove `enforce_file_num` Function**: Deleted the `enforce_file_num` function and its related test cases in `twcs.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Testing - **Add `merge_seq_files` Functionality**: Implemented the `merge_seq_files` function in `run.rs` to optimize file merging based on scoring systems. Updated benchmarks in `run_bench.rs` to include `bench_merge_seq_files`. - **Improve Compaction Strategy in `twcs.rs`**: Modified the compaction logic to handle file merging more effectively, considering file size and overlap. - **Update Tests**: Enhanced test coverage in `compaction_test.rs` and `append_mode_test.rs` to validate new compaction logic and file merging strategies. - **Remove Unused Function**: Deleted `new_file_handles` from `test_util.rs` as it was no longer needed. * feat/multiple-compaction-output: ### Refactor TWCS Compaction Options - **Refactor Compaction Logic**: Simplified the TWCS compaction logic by replacing multiple parameters (`max_active_window_runs`, `max_active_window_files`, `max_inactive_window_runs`, `max_inactive_window_files`) with a single `trigger_file_num` parameter in `picker.rs`, `twcs.rs`, and `options.rs`. - **Update Tests**: Adjusted test cases to reflect the new compaction logic in `append_mode_test.rs`, `compaction_test.rs`, `filter_deleted_test.rs`, `merge_mode_test.rs`, and various test files under `tests/cases`. - **Modify Engine Options**: Updated engine option keys to use `trigger_file_num` in `mito_engine_options.rs` and `region_request.rs`. - **Fuzz Testing**: Updated fuzz test generators and translators to accommodate the new compaction parameter in `alter_expr.rs` and related files. This refactor aims to streamline the compaction configuration by reducing the number of parameters and simplifying the codebase. * chore: add trailing space * fix license header * feat/revise-compaction-picker: **Limit File Processing and Optimize Merge Logic in `run.rs`** - Introduced a limit to process a maximum of 100 files in `merge_seq_files` to control time complexity. - Adjusted logic to calculate `target_size` and iterate over files using the limited set of files. - Updated scoring calculations to use the limited file set, ensuring efficient file merging. * feat/revise-compaction-picker: ### Add Compaction Metrics and Remove Debug Logging - **Compaction Metrics**: Introduced new histograms `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` to track compaction input and output file sizes in `metrics.rs`. Updated `compactor.rs` to observe these metrics during the compaction process. - **Logging Cleanup**: Removed debug logging of file ranges during the merge process in `twcs.rs`. * feat/revise-compaction-picker: ## Enhance Compaction Logic and Metrics - **Compaction Logic Improvements**: - Added methods `input_file_size` and `output_file_size` to `MergeOutput` in `compactor.rs` to streamline file size calculations. - Updated `Compactor` implementation to use these methods for metrics tracking. - Modified `Ranged` trait logic in `run.rs` to improve range comparison. - Enhanced test cases in `run.rs` to reflect changes in compaction logic. - **Metrics Enhancements**: - Changed `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` from histograms to counters in `metrics.rs` for better performance tracking. - **Debugging and Logging**: - Added detailed logging for compaction pick results in `twcs.rs`. - Implemented custom `Debug` trait for `FileMeta` in `file.rs` to improve debugging output. - **Testing Enhancements**: - Added new test `test_compaction_overlapping_files` in `compaction_test.rs` to verify compaction behavior with overlapping files. - Updated `merge_mode_test.rs` to reflect changes in file handling during scans. * feat/revise-compaction-picker: ### Update `FileHandle` Debug Implementation - **Refactor Debug Output**: Simplified the `fmt::Debug` implementation for `FileHandle` in `src/mito2/src/sst/file.rs` by consolidating multiple fields into a single `meta` field using `meta_ref()`. - **Atomic Operations**: Updated the `deleted` field to use atomic loading with `Ordering::Relaxed`. * Trigger CI * feat/revise-compaction-picker: **Update compaction logic and default options** - **`twcs.rs`**: Enhanced logging for compaction pick results by improving the formatting for better readability. - **`options.rs`**: Modified the default `max_output_file_size` in `TwcsOptions` from 2GB to 512MB to optimize file handling and performance. * feat/revise-compaction-picker: Refactor `find_overlapping_items` to use an external result vector - Updated `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to accept a mutable result vector instead of returning a new vector, improving memory efficiency. - Modified benchmarks in `src/mito2/benches/bench_compaction_picker.rs` to accommodate the new function signature. - Adjusted tests in `src/mito2/src/compaction/run.rs` to use the updated function signature, ensuring correct functionality with the new approach. * feat/revise-compaction-picker: Improve file merging logic in `run.rs` - Refactor the loop logic in `merge_seq_files` to simplify the iteration over file groups. - Adjust the range for `end_idx` to include the endpoint, allowing for more flexible group selection. - Remove the condition that skips groups with only one file, enabling more comprehensive processing of file sequences. * feat/revise-compaction-picker: Enhance `find_overlapping_items` with `SortedRun` and Update Tests - Refactor `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to utilize the `SortedRun` struct for improved efficiency and clarity. - Introduce a `sorted` flag in `SortedRun` to optimize sorting operations. - Update test cases in `src/mito2/benches/bench_compaction_picker.rs` to accommodate changes in `find_overlapping_items` by using `SortedRun`. - Add `From<Vec<T>>` implementation for `SortedRun` to facilitate easy conversion from vectors. * feat/revise-compaction-picker: **Enhancements in `compaction/run.rs`:** - Added `ReadableSize` import to handle size calculations. - Modified the logic in `merge_seq_files` to clamp the calculated target size to a maximum of 2GB when `max_file_size` is not provided. * feat/revise-compaction-picker: Add Default Max Output Size Constant for Compaction Introduce DEFAULT_MAX_OUTPUT_SIZE constant to define the default maximum compaction output file size as 2GB. Refactor the merge_seq_files function to utilize this constant, ensuring consistent and maintainable code for handling file size limits during compaction. |
||
 Lei, HUANGandLei
|
5a9023d6b3 |
feat(bulk): write to multiple time partitions (#6086)
* add benchmark for splitting according to time partition * feat/write-to-multiple-time-partitions: **Enhancements to Bulk Processing and Time Partitioning** - **`part.rs`**: Added `Snafu` to imports and introduced `timestamp_index` in `BulkPart` struct. Implemented `timestamps` method for accessing timestamp columns. - **`simple_bulk_memtable.rs`**: Updated tests to include `timestamp_index` initialization. - **`time_partition.rs`**: Enhanced `TimePartition` to support partial writes with `write_record_batch_partial`. Implemented `split_record_batch` for filtering records by timestamp range. Added comprehensive tests for `split_record_batch`. - **`handle_bulk_insert.rs`**: Modified to retrieve timestamp index and column together, updating `BulkPart` initialization with `timestamp_index`. * feat/write-to-multiple-time-partitions: ### Enhance Time Partitioning Logic - **`time_partition.rs`**: - Introduced `HashSet` for efficient partition management. - Refactored `write_bulk` to handle multiple partitions and added `find_partitions_by_time_range` for identifying existing and missing partitions. - Updated `get_or_create_time_partition` to manage partition creation. - Added comprehensive tests for partition finding logic, covering various scenarios including overlapping and non-overlapping time ranges. - **Tests**: - Added `test_find_partitions_by_time_range` to validate new partitioning logic. - Updated `test_split_record_batch` to ensure correct record batch splitting behavior. * feat/write-to-multiple-time-partitions: ### Enhance Time Partitioning and Testing in `time_partition.rs` - **Time Partitioning Enhancements**: - Updated `split_record_batch` to handle multiple timestamp units (`Second`, `Millisecond`, `Microsecond`, `Nanosecond`) by matching on `DataType`. - Improved filtering logic for timestamp arrays to support various time units. - **Testing Enhancements**: - Added `test_write_bulk` to verify writing across multiple partitions and scenarios in `time_partition.rs`. - Updated `test_split_record_batch` to use `TimestampMillisecondArray` for testing timestamp partitioning. - **Imports and Dependencies**: - Added necessary imports for new timestamp array types and testing utilities. * feat/write-to-multiple-time-partitions: ### Refactor and Enhance Time Partition Filtering - **Refactor Filtering Logic**: Consolidated the filtering logic for timestamp arrays using macros in `time_partition.rs` and `bench_filter_time_partition.rs`. This reduces code duplication and improves maintainability. - **Enhance `BulkPart` Struct**: Made fields in `BulkPart` public to facilitate easier access and manipulation in `memtable.rs` and `part.rs`. - **Rename Function**: Renamed `split_record_batch` to `filter_record_batch` for clarity in `time_partition.rs` and `bench_filter_time_partition.rs`. - **Add Feature Flag**: Introduced `int_roundings` feature in `lib.rs` to support new functionality. * refactor tests * feat/write-to-multiple-time-partitions: Improve timestamp handling in `time_partition.rs` - Enhanced safety comments for timestamp conversion to ensure clarity. - Modified logic to prevent overflow by using `div_euclid` for `bulk_start_sec` and `bulk_end_sec` calculations. - Adjusted the `filter_map` logic to correctly compute timestamps using `start_sec` and `part_duration_sec`. * feat/write-to-multiple-time-partitions: **Refactor timestamp handling and add utility function** - **Refactor `time_partition.rs`:** Simplified timestamp handling by replacing direct type access with a utility function to retrieve the timestamp unit. Improved error handling for timestamp conversion. - **Enhance `metadata.rs`:** Added `time_index_type` function to `RegionMetadata` to retrieve the timestamp type of the time index column, ensuring safer and more readable code. * feat/write-to-multiple-time-partitions: Refactor time partition variable names in `time_partition.rs` - Renamed variables for clarity: `bulk_start_sec` to `start_bucket` and `bulk_end_sec` to `end_bucket`. - Updated related logic to use new variable names for improved readability and maintainability. * feat/write-to-multiple-time-partitions: **Refactor variable names in `time_partition.rs`** - Updated variable names from `matching` and `missing` to `matchings` and `missings` for clarity and consistency. - Modified function calls and loop iterations to align with the new variable names. - Affected file: `src/mito2/src/memtable/time_partition.rs` * feat/write-to-multiple-time-partitions: ### Refactor variable names in `time_partition.rs` - Updated variable names for clarity in `time_partition.rs`: - Renamed `matchings` to `matching_parts` - Renamed `missings` to `missing_parts` - Adjusted logic to use new variable names in methods `find_partitions_by_time_range` and `write_record_batch`. * feat/write-to-multiple-time-partitions: ### Enhance Time Partition Handling - **`time_partition.rs`**: - Added `ArrayRef` to handle timestamp arrays, improving the partitioning logic by allowing more efficient timestamp range checks. - Enhanced `find_partitions_by_time_range` to support sparse data and handle different timestamp units (`Second`, `Millisecond`, `Microsecond`, `Nanosecond`). - Updated test cases to cover new scenarios, including sparse data and edge cases, ensuring robustness of partition handling. --------- Co-authored-by: Lei <lei@Leis-MacBook-Pro.local> |
||
|
fys
|
2b2ea5bf72 |
chore: upgrade some dependencies (#5777)
* chore: upgrade some dependencies * chore: upgrade some dependencies * fix: cr * fix: ci * fix: test * fix: cargo fmt |
||
 Weny Xu
|
965a48656f |
feat(metric-engine): introduce RowModifier for MetricEngine (#5380)
* feat(metric-engine): store physical table ColumnIds in `MetricEngineState` * feat(metric-engine): introduce `RowModifier` for MetricEngine * chore: upgrade greptime-proto * feat: introduce `WriteHint` to `RegionPutRequest` * chore: apply suggestions from CR * chore: udpate greptime-proto * chore: apply suggestions from CR * chore: add comments * chore: update proto |
||
 discord9
|
758aef39d8 |
feat: filter batch by sequence in memtable (#5367)
* feat: add seq field * feat: filter by sequence * chore: per review * docs: explain why not prune * chore: correct doc * test: test filter by seq |
||
|
Weny Xu
|
b64c075cdb |
feat: introduce PrimaryKeyEncoding (#5312)
* feat: introduce `PrimaryKeyEncoding` * fix: fix unit tests * chore: add empty line * test: add unit tests * chore: fmt code * refactor: introduce new codec trait to support various encoding * fix: fix unit tests * chore: update sqlness result * chore: apply suggestions from CR * chore: apply suggestions from CR |
||
 YingwenandJeremyhi
|
10b7a3d24d |
feat: Implements merge_mode region options (#4208)
* feat: add update_mode to region options * test: add test * feat: last not null iter * feat: time series last not null * feat: partition tree update mode * feat: partition tree * fix: last not null iter slice * test: add test for compaction * test: use second resolution * style: fix clippy * chore: merge two lines Co-authored-by: Jeremyhi <jiachun_feng@proton.me> * chore: address CR comments * refactor: UpdateMode -> MergeMode * refactor: LastNotNull -> LastNonNull * chore: return None earlier * feat: validate region options make merge mode optional and use default while it is None * test: fix tests --------- Co-authored-by: Jeremyhi <jiachun_feng@proton.me> |
||
 maco
|
40c585890a |
refactor: replace Expr with datafusion::Expr (#3995)
* refactor: replace Expr with datafusion::Expr * fix: fmt-toml * fix: cr comment |
||
|
Yingwen
|
39b69f1e3b |
refactor!: Renames the new memtable to PartitionTreeMemtable (#3547)
* refactor: rename mod merge_tree to partition_tree * refactor: rename merge_tree * refactor: change merge tree comment * refactor: rename merge tree struct * refactor: memtable options |
||
|
Lei, HUANG
|
ddbcff68dd |
feat: support append-only mode in time-series memtable (#3540)
* feat: support append-only mode in time-series memtable * fix: rename sort_and_dedup to sort |
||
|
Yingwen
|
7c895e2605 |
perf: more benchmarks for memtables (#3491)
* chore: remove duplicate bench * refactor: rename bench * perf: add full scan bench for memtable * feat: filter bench and add time series to bench group * chore: comment * refactor: rename * style: fix clippy |
||
|
Lei, HUANG
|
376409b857 |
feat: employ sparse key encoding for shard lookup (#3410)
* feat: employ short key encoding for shard lookup * fix: license * chore: simplify code * refactor: only enable sparse encoding to speed lookup on metric engine * fix: names |