Lei, HUANG ab21f684bf perf(mito2): lazily materialize sparse primary key tags (#9199) ...

* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations

The new revision (v0y4g3r/memcomparable main, PR #1) includes:
- Deserializer::read_bytes_into for reusable-buffer decoding
- read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain)
- bulk put_slice in both normal and reverse modes
- Error::Eof instead of panic on truncated input

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): extract sparse tag values lazily without full PK decode

Replace the full primary-key decode in two hot consumers of
SortField::deserialize with column-lazy extraction over
SparsePrimaryKeyView, eliminating the per-label owned values
(Vec<Value>, one heap allocation per string) for every distinct key:

- flat_format: DecodedPrimaryKeys keeps sparse keys encoded and
  extracts each tag column directly from the raw dictionary bytes into
  a reusable buffer. FlatConvertFormat (flush/compaction write path)
  builds all projected tag columns in one pass per key, sharing offset
  discovery between columns; the read path (file_range filters) keeps
  per-column lazy extraction.
- series_index writer: builds rows from the reserved prefix and only
  the indexed tags via a shared offsets cache and buffer, instead of
  decoding all labels and copying out a few strings.
- mito-codec: add SparsePrimaryKeyView::reserved_value for typed
  table_id/tsid extraction.

Persisted formats are unchanged; values pushed to column builders are
identical (NULL vs empty string, reserved column types, duplicate
dictionary entries and consecutive-run order are preserved).

Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion
30 samples): old = full decode + per-column builds, new = lazy
extraction (+ one-pass for multi-column projection).

| Workload | Before | After | Change |
| --- | ---: | ---: | ---: |
| 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% |
| 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% |
| 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% |
| 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% |
| 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% |

Also adds a decode_sparse benchmark group to mito-codec's
bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): share sparse tag scans across filters

Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics.

Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches.

Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): benchmark sparse tag predicate count scaling

Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

2026-09-17 10:49:19 +00:00

..

bench_primary_key_filter.rs

refactor(mito2)!: remove PartitionTreeMemtable (#8080)

2026-05-15 11:49:27 +00:00

bench_sparse_encoding.rs

perf(mito2): lazily materialize sparse primary key tags (#9199)

2026-09-17 10:49:19 +00:00