mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-20 17:54:12 +00:00
7309800aef64f29548268fc43aff3a0eb50adfc7
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ab21f684bf |
perf(mito2): lazily materialize sparse primary key tags (#9199)
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations The new revision (v0y4g3r/memcomparable main, PR #1) includes: - Deserializer::read_bytes_into for reusable-buffer decoding - read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain) - bulk put_slice in both normal and reverse modes - Error::Eof instead of panic on truncated input Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): extract sparse tag values lazily without full PK decode Replace the full primary-key decode in two hot consumers of SortField::deserialize with column-lazy extraction over SparsePrimaryKeyView, eliminating the per-label owned values (Vec<Value>, one heap allocation per string) for every distinct key: - flat_format: DecodedPrimaryKeys keeps sparse keys encoded and extracts each tag column directly from the raw dictionary bytes into a reusable buffer. FlatConvertFormat (flush/compaction write path) builds all projected tag columns in one pass per key, sharing offset discovery between columns; the read path (file_range filters) keeps per-column lazy extraction. - series_index writer: builds rows from the reserved prefix and only the indexed tags via a shared offsets cache and buffer, instead of decoding all labels and copying out a few strings. - mito-codec: add SparsePrimaryKeyView::reserved_value for typed table_id/tsid extraction. Persisted formats are unchanged; values pushed to column builders are identical (NULL vs empty string, reserved column types, duplicate dictionary entries and consecutive-run order are preserved). Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion 30 samples): old = full decode + per-column builds, new = lazy extraction (+ one-pass for multi-column projection). | Workload | Before | After | Change | | --- | ---: | ---: | ---: | | 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% | | 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% | | 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% | | 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% | | 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% | Also adds a decode_sparse benchmark group to mito-codec's bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): share sparse tag scans across filters Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics. Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches. Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): benchmark sparse tag predicate count scaling Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
7840aa1bb4 |
refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk Signed-off-by: evenyag <realevenyag@gmail.com> * chore: keep partition tree memtable for migration test Restore PartitionTreeMemtable construction when memtable.type=partition_tree is explicit, and move the sparse-encoding bulk override into the default (no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse, so the test actually exercises a PartitionTreeMemtable in phase 1 and verifies WAL replay into the new BulkMemtable on reopen without flushing. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: drop partition tree memtable from runtime Re-apply the unconditional sparse-encoding override in `MemtableBuilderProvider::builder_for_options` and route the `MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation warning. After this change, `PartitionTreeMemtableBuilder` is no longer reachable from the engine runtime; benchmarks still reference the type. Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the `put_sparse_rows` helper added in the previous commit — that test only existed to validate the PartitionTree -> Bulk reopen migration and is unnecessary now that the override is in place. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): move timestamp_array_to_i64_slice into read module Relocate the timestamp_array_to_i64_slice helper from memtable/partition_tree/data.rs to the read module so that the read path no longer depends on the partition_tree internals. All call sites (both inside and outside the partition_tree module) now import from crate::read. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests The time_partition tests use the memtable builder purely as a generic backend for the TimePartitions write/scan paths; nothing in them is specific to the partition-tree memtable. Switch the seven affected tests to TimeSeriesMemtableBuilder so the tests no longer depend on PartitionTreeMemtableBuilder. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): delete PartitionTreeMemtable implementation The runtime already falls back to BulkMemtable for the PartitionTree variant. Drop the now-unreachable implementation, its metrics, the partition_tree benchmarks, the metric-engine Unsupported fallback in bulk_insert.rs, and the test helpers that only existed for the deleted module. MemtableOptions::PartitionTree, its parsing, the runtime fallback, the store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures remain so existing region options keep round-tripping. Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito-codec): drop skip_partition_column parameter PartitionTreeMemtable was the only caller passing skip_partition_column=true; every other caller passes false. Now that the partition_tree module is gone, the parameter is uniformly false and the guard branch is dead. Drop the parameter from the trait method and both impls, remove the guard and the is_partition_column helper, and update the four remaining call sites in mito2 plus the bench. Signed-off-by: evenyag <realevenyag@gmail.com> * chore(mito2): remove unused MemtableConfig enum Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused variant Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test_config_api Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove unused memtable test helpers Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: support bulk memtable options Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sanitize config Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove partition tree options from region options Move primary_key_encoding to the top level Signed-off-by: evenyag <realevenyag@gmail.com> * test: make ssts test datetime replaced text stable Signed-off-by: evenyag <realevenyag@gmail.com> * test: update sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * chore: validate_enum_options consider bulk memtable Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass region id when parsing region options Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with `try_from_options(region_id, options_map)` so the legacy partition_tree fallback can log the affected region. The fallback now also overrides the SST format to flat in addition to clearing the memtable type. Signed-off-by: evenyag <realevenyag@gmail.com> * fix: align sst_format with bulk memtable on parse and open Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
e1156728fc |
perf(mito-codec): optimize SparseValues decode and lookup (#8057)
* perf: add benchmarks for SparsePrimaryKeyCodec::has_column Add benchmarks covering table_id, tsid, first_tag, and last_tag lookups across 5, 10, 50, and 100 tag counts to measure the cost of the offset map construction in has_column. Signed-off-by: evenyag <realevenyag@gmail.com> * perf: handle table id/tsid specially Signed-off-by: evenyag <realevenyag@gmail.com> * perf: lazy decode Signed-off-by: evenyag <realevenyag@gmail.com> * chore: use vec for small tags Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add bench Signed-off-by: evenyag <realevenyag@gmail.com> * perf: use 32 as inline capacity Signed-off-by: evenyag <realevenyag@gmail.com> * perf: benchmark sparse value Signed-off-by: evenyag <realevenyag@gmail.com> * feat: change sparse values to use vec Signed-off-by: evenyag <realevenyag@gmail.com> * chore: reserve capacity Signed-off-by: evenyag <realevenyag@gmail.com> * chore: simplify comments Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update benchmark for map sparse values Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove empty check Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
92e2d71f48 |
feat: implement prefilter framework and primary key prefilter (#7862)
* feat: prefilter basic framework Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: move arguments to RowGroupBuildContext Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: skip prefiltered exprs in FlatPruneReader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused functions Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * feat: handle partition columns in prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: apply prefiltered selection by and_then Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: handle last row cache Signed-off-by: evenyag <realevenyag@gmail.com> * fix: don't ignore error in PrimaryKeyFilter Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
3ce7e7bc81 |
perf: accelerate pk filter (#7584)
* perf: accelerate pk filter Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add benchmark Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * unit tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * support compare ops Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor to reduce dense decode/encode duplication Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor if-else chain Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
749a5ab165 |
feat: struct value and vector (#7033)
* feat: struct value Signed-off-by: Ning Sun <sunning@greptime.com> * feat: update for proto module * feat: wip struct type * feat: implement more vector operations * feat: make datatype and api * feat: reoslve some compilation issues * feat: resolve all compilation issues * chore: format update * test: resolve tests * test: test and refactor value-to-pb * feat: add more tests and fix for value types * chore: remove dbg * feat: test and fix iterator * fix: resolve struct_type issue * refactor: use vec for struct items * chore: update proto to main branch * refactor: address some of review issues * refactor: update for further review * Add validation on new methods * feat: update struct/list json serialization * refactor: reimplement get in struct_vector * refactor: struct vector functions * refactor: fix lint issue * refactor: address review comments --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
|
c9377e7c5a |
build: bump rust edition to 2024 (#6920)
* bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
dd3432e6ca |
chore: change encode raw values signature (#6869)
* chore/change-encode-raw-values-sig: ### Update Sparse Encoding to Use Byte Slices - **`bench_sparse_encoding.rs`**: Modified the `encode_raw_tag_value` function to use byte slices instead of `Bytes` for tag values. - **`sparse.rs`**: Updated the `encode_raw_tag_value` method in `SparsePrimaryKeyCodec` to accept byte slices (`&[u8]`) instead of `Bytes`. Adjusted related test cases to reflect this change. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/change-encode-raw-values-sig: ### Add `Clear` Trait Implementation for Byte Slices - Implemented the `Clear` trait for byte slices (`&[u8]`) in `repeated_field.rs` to enhance trait coverage and provide a default clear operation for byte slice types. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
|
d99734b97b |
perf: sparse encoder (#6809)
* perf/sparse-encoder: - **Update Dependencies**: Updated `criterion-plot` to version `0.5.0` and added `criterion` version `0.7.0` in `Cargo.lock`. Added `bytes` to `Cargo.toml` in `src/metric-engine`. - **Benchmarking**: Added a new benchmark for sparse encoding in `bench_sparse_encoding.rs` and updated `Cargo.toml` in `src/mito-codec` to include `criterion` as a dev-dependency. - **Sparse Encoding Enhancements**: Modified `SparsePrimaryKeyCodec` in `sparse.rs` to include new methods `encode_raw_tag_value` and `encode_internal`. Added public constants `RESERVED_COLUMN_ID_TSID` and `RESERVED_COLUMN_ID_TABLE_ID`. - **HTTP Server**: Made `try_decompress` function public in `prom_store.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/sparse-encoder: Improve buffer handling in `sparse.rs` - Refactored buffer reservation logic to use `value_len` for clarity. - Optimized chunk processing by calculating `num_chunks` and `remainder` for efficient data handling. - Enhanced manual serialization of bytes to avoid byte-by-byte operations, improving performance. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Update src/mito-codec/src/row_converter/sparse.rs Co-authored-by: Yingwen <realevenyag@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |