Commit Graph
35 Commits
Author SHA1 Message Date
discord9 7309800aef feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema

A raw series frame written with key-encoded labels carries the primary key
as opaque bytes; only the binding held by the scanner says how to turn
those bytes back into a metric name and labels. A consumer that receives
the frame without that binding cannot decode it, so the schema now
describes its own label layout.

The description is compact JSON under `greptime.series.label_decode`,
carrying a version, the metric name, the primary-key encoding, the ordered
key columns with their physical types, and the logical tags to expose
-- exactly what decoding reads. It is derived from the binding, so there
is one writer, and a reader that does not implement the version it finds
rejects the frame instead of guessing.

`unpack_raw_series` keeps its binding argument and now requires it to
agree with the description the schema declares, so the two cannot drift.
`PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key
layout, so the metadata-backed and description-backed paths share their
validation instead of growing a second decoder.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datatypes): compare label descriptions by what their order means

The frame's declared label-decode description was compared to the binding's with
plain equality, but the description has one part whose order carries no meaning:
the decoder sorts logical tags by name when it binds them, so the same tags listed
in another order denote the same labels and were being refused.

`same_label_layout` canonicalises exactly that part and leaves `primary_key` in
place, because for a dense key the array position is the encoding position: the same
columns in another order are a different key, not the same key reordered. Both
fields now say which order they mean.

Three checks go with it. The binding stays for a stated reason -- it establishes
identity, the physical region key layout a decoded identity came from -- rather than
because a count of call sites kept it. The post-pack schema comparison in the series
scan could not fire: the leaf already re-derives that schema and compares it, and
every `SeriesSchema` field is covered by the checks between them. And two test
groups re-asserted what the tracked v1_metadata fixture pins case by case.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(promql): decode a raw frame from the frame alone

`unpack_raw_series` took the label mode, the ordering and the caller's binding
along with the stream, so a consumer that received a frame from a peer still
needed a local copy of the binding to read it — the thing the frame's own
description exists to avoid, and the reason the description was freezing a wire
format for nobody.

The stream is now the only input: the field schema, the label mode, the label
decode description and the producer ordering all come from the frame's schema,
and every frame is validated whether or not a caller asked for it. `expected`
stays as an optional check on the schema that is declared — field names and their
order, and no evaluation grid — and supplies nothing to the decoder, so a caller
cannot smuggle in a layout the frame does not declare.

Three call sites compared the caller's binding against the frame and all three
were load-bearing: a peer's response is not trusted to be the region that was
planned, and a native plan's child is an arbitrary execution plan. Those became
`ensure_frame_matches_label_source`, which states what it compares: the label
layout, not the region. Two regions with the same tag ids and names compare equal,
and the doc now says so instead of claiming to establish identity.

A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers
`label_decode_spec()` instead; every consumer that decides whether to decode reads
the description, so the placeholder never stands in for "these labels are
complete".

The merge contract is the description, not the binding: inputs are compared by
the layout they declare, and a binding that could not be expressed on the wire — a
`RegionMetadata` snapshot — no longer makes the merge stricter than the frames it
merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a
copy, because the caller's binding keeps its own order and the table scan compares
producer tag lists positionally, and readers compare by equality.

The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to
cover. Rather than mirror all 25 datatype variants into a type this module owns, a
frozen fixture records the text of every variant and an exhaustive match makes a
new variant fail to compile until it is recorded. The version policy is written
next to the constant: exact-text comparison across nodes, unknown versions
rejected, any same-version change breaking — with the planning-time fallback
stated as intended and marked not implemented.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(store-api): make every key-encoded raw stream declare its layout

A raw series stream whose chunks carry key-encoded labels had to be decoded
against a scan-bound `RawLabelSource`, and two consumers compensated for
producers that stayed silent by deriving a description from that binding:
`pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels`
did not compensate, so an unframed producer stream reached it with no
description and a chunk carrying no labels, and failed.

The stream itself is the authority for its chunks. A key-encoded producer
knows its own region metadata and can state the layout it writes, exactly as a
frame states the layout it carries, so the split contract — "a producer
declares its identity, a frame declares its layout" — is replaced by one rule:
a stream whose chunks carry key-encoded labels declares the description, and a
stream whose chunks carry complete labels answers `None`.

mito2's `RawStream` caches the derivation at construction, so the declaration
exists before the first poll and for an empty stream. The trait documentation
states the rule and no longer exempts unframed producers. Both fallbacks are
gone: `pack_raw_series` reads the stream's declaration only, and rejects a
key-encoded stream that declares none rather than packing it as an
explicit-label frame, which would describe chunks carrying no labels as if they
carried all of them. The merge reads the declaration only.

Test fixtures that model key-encoded producers now declare their layout too;
fixtures that carry complete labels keep answering `None`. The new mito2 test
decodes a real produced key through the declared description for both dense and
sparse encodings, and checks the declaration is present before polling even for
an empty stream.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(common-query): let the declaration decide the frame mode alone

`pack_raw_series` rejected a key-encoded stream that declared no description
outright, so a stream that stays silent could not be packed at all. That check
was redundant: the declaration already decides the frame's label mode, and the
first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor`
with "raw chunk label mode does not match frame mode", so no such stream can
pack data either way.

What the check added was an early error for a producer that already breaks the
contract, in exchange for reading the stream's scan binding here — in the one
function whose rule is that the declaration is the only decode input. Drop it,
so pack keeps deciding the mode from the declaration and nothing else.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(store-api): assert the fixture's declared label decode description

The mock prepared source derives a `LabelDecodeSpec` from its label source and
reports it through `RawSeriesStream::label_decode_spec`, but no assertion read
it, so the derivation could drift from the chunks the fixture actually builds.
Check it against the description the fixture's own key columns imply: an
explicit source declares nothing, a primary-key source declares the metric, the
dense encoding, the one key column and its tag.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): let the frame's declaration be the only label authority

Two things still treated a caller-supplied binding as authoritative after the
declaration became the decode input:

- A merged stream forwarded its first input's scan-bound identity, though a
  merge is not a region scan and decodes from the declaration alone. It now
  reports no bound identity.
- `KeyColumn.tag` repeated an invariant every key column is a tag. The check
  moves to the one place a decoder is bound, so a snapshot built without
  validation is still rejected instead of silently decoded.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-21 19:11:24 +08:00
discord9 410759ed5a feat(mito-codec): decode scan-bound primary key labels
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 10:35:06 +08:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
Lei, HUANG e91faa9df8 perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 15:52:54 +00:00
Lei, HUANG e9dd79d1d2 fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions

Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.

Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): report invalid primary key ranges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): scope primary key ranges to comparisons

Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.

Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.

Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): cache aligned primary key ranges per file

Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.

Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify primary key mapper schema snapshot

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): align primary key range fixtures with table contract

Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.

Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): enable filesystem GC in release unit tests

Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito-codec): skip release value decoding in PK prefix counts

Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.

Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito-codec): remove redundant prefix integration tests

Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: wait for MySQL to accept authenticated TCP queries

Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.

Verify readiness with SELECT 1 over TCP rather than the initialization socket.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-20 14:30:12 +00:00
Lei, HUANG cb6c7a5c6d perf(mito-codec): streamline sparse label extraction and buffer sizing (#9217)
* perf(mito-codec): validate sparse labels once

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito-codec): reserve complete sparse label chunks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-18 09:51:01 +00:00
Lei, HUANG ab21f684bf perf(mito2): lazily materialize sparse primary key tags (#9199)
* chore: bump memcomparable to d8fb3558 for bytes I/O optimizations

The new revision (v0y4g3r/memcomparable main, PR #1) includes:
- Deserializer::read_bytes_into for reusable-buffer decoding
- read_bytes 32B pre-allocation (no more 0→8→16→32 realloc chain)
- bulk put_slice in both normal and reverse modes
- Error::Eof instead of panic on truncated input

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): extract sparse tag values lazily without full PK decode

Replace the full primary-key decode in two hot consumers of
SortField::deserialize with column-lazy extraction over
SparsePrimaryKeyView, eliminating the per-label owned values
(Vec<Value>, one heap allocation per string) for every distinct key:

- flat_format: DecodedPrimaryKeys keeps sparse keys encoded and
  extracts each tag column directly from the raw dictionary bytes into
  a reusable buffer. FlatConvertFormat (flush/compaction write path)
  builds all projected tag columns in one pass per key, sharing offset
  discovery between columns; the read path (file_range filters) keeps
  per-column lazy extraction.
- series_index writer: builds rows from the reserved prefix and only
  the indexed tags via a shared offsets cache and buffer, instead of
  decoding all labels and copying out a few strings.
- mito-codec: add SparsePrimaryKeyView::reserved_value for typed
  table_id/tsid extraction.

Persisted formats are unchanged; values pushed to column builders are
identical (NULL vs empty string, reserved column types, duplicate
dictionary entries and consecutive-run order are preserved).

Benchmark (bench_pk_tag_column, 4096 rows, 40 labels per key, Criterion
30 samples): old = full decode + per-column builds, new = lazy
extraction (+ one-pass for multi-column projection).

| Workload | Before | After | Change |
| --- | ---: | ---: | ---: |
| 1 of 40 tags, 4096 distinct keys | 6.95 ms | 803 us | -88% |
| 1 of 40 tags, 128 distinct keys | 206.8 us | 29.2 us | -86% |
| 10 of 40 tags, 128 distinct keys | 236.2 us | 75.5 us | -68% |
| 40 of 40 tags, 128 distinct keys | 314 us | 215.2 us | -31% |
| 1 of 10 tags, 4096 distinct keys | 1.75 ms | 207.6 us | -88% |

Also adds a decode_sparse benchmark group to mito-codec's
bench_sparse_encoding covering 10/40 labels x 10/24/64-byte values.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): share sparse tag scans across filters

Batch missing tag columns used by simple filters and partition pruning, while retaining single-column extraction and skip-mode semantics.

Replace the optional reserved-byte accessor with typed table_id/tsid reads from the validated prefix. Add full-decode equivalence tests for sliced dictionaries, NULL/empty labels and cross-key scratch reuse, plus series-index roundtrips across batches.

Benchmark real precise filtering with 4096 rows: 10 tag predicates drop from 307.85 to 162.49 us; 40 predicates drop from 894.03 to 550.55 us. All 1459 mito2/mito-codec tests pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): benchmark sparse tag predicate count scaling

Add a dedicated precise-filter benchmark group for 1, 2, 4, 8, 16, and 32 predicates with both 1 and 32 rows per primary key. Keep 4096 rows, 40 labels, and full selectivity constant across cases, and validate the workload outside the timed section.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-17 10:49:19 +00:00
Lei, HUANG d6a8974613 perf(mito2): optimize materialized column index updates (#9182)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-16 16:28:12 +00:00
Lei, HUANG 7aacc947b3 perf(mito2): lazily extract sparse primary key index values (#9176)
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-16 06:15:52 +00:00
Yingwen bfd7abb0b8 feat(mito): add candidate series scanner (#8586)
* feat(mito): add candidate series scanner

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): exclude other ranges from candidate scan

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(mito): bench series candidate scan

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): clean up series candidate scanner

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): improve candidate series scan

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): guard primary-key-only SST reads

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs(mito2): clarify candidate scanner invariants

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): compat primary-key-only reads

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-07-23 07:08:03 +00:00
Yingwen 7840aa1bb4 refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: keep partition tree memtable for migration test

Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: drop partition tree memtable from runtime

Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.

Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): move timestamp_array_to_i64_slice into read module

Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests

The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(mito2): delete PartitionTreeMemtable implementation

The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.

MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito-codec): drop skip_partition_column parameter

PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(mito2): remove unused MemtableConfig enum

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt code

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove unused variant

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update test_config_api

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: remove unused memtable test helpers

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address review comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: support bulk memtable options

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: sanitize config

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: remove partition tree options from region options

Move primary_key_encoding to the top level

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: make ssts test datetime replaced text stable

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update sqlness result

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: validate_enum_options consider bulk memtable

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: pass region id when parsing region options

Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: align sst_format with bulk memtable on parse and open

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-15 11:49:27 +00:00
Yingwen e1156728fc perf(mito-codec): optimize SparseValues decode and lookup (#8057)
* perf: add benchmarks for SparsePrimaryKeyCodec::has_column

Add benchmarks covering table_id, tsid, first_tag, and last_tag
lookups across 5, 10, 50, and 100 tag counts to measure the cost of
the offset map construction in has_column.

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: handle table id/tsid specially

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: lazy decode

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: use vec for small tags

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add bench

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: use 32 as inline capacity

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: benchmark sparse value

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: change sparse values to use vec

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: reserve capacity

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: simplify comments

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update benchmark for map sparse values

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove empty check

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-08 08:35:34 +00:00
Yingwen 555741a9f1 feat: prune bulk memtable parts by first tag (#7911)
* feat: initial implementation for MultiBulkPart minmax pruning

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: add debug logs

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: prune bulk part

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt code

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address PR review comments for bulk part pruning

Document sparse encoding format in SparsePrimaryKeyCodec and add
comment explaining why primary_key.first() works for both encodings.
Remove noisy info-level pruning logs from the read path.

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add test for MultiBulkPart batch pruning behavior

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: don't return error when read() returns None

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address review comments

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-04-21 10:08:16 +00:00
shuiyisong ba32c5fe9e chore: remove unused deps using udeps (#7906)
* chore: remove unused deps using udeps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fmt toml

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-04-02 06:49:27 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
Yingwen 92e2d71f48 feat: implement prefilter framework and primary key prefilter (#7862)
* feat: prefilter basic framework

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: move arguments to RowGroupBuildContext

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: skip prefiltered exprs in FlatPruneReader

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove unused functions

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: handle partition columns in prefilter

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: apply prefiltered selection by and_then

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: handle last row cache

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: don't ignore error in PrimaryKeyFilter

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-03-30 04:24:20 +00:00
Ruihang Xia 3ce7e7bc81 perf: accelerate pk filter (#7584)
* perf: accelerate pk filter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* unit tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* support compare ops

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor to reduce dense decode/encode duplication

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor if-else chain

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-19 06:41:51 +00:00
Yingwen fed6cb0806 fix: flat format use correct encoding in indexer for tags (#7440)
* test: add inverted and skipping test

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: Add tests for fulltext index

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: index dictionary type in correct encoding in flat format

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: use encode_data_type() in SortField

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: refine imports

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add tests for sparse encoding

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove logs

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update list test

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: simplify tests

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-12-19 07:36:44 +00:00
Ruihang Xia 5bf72ab327 feat: decode_primary_key method for debugging (#7284)
* initial impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* third param

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: support convert Dictionary type to ConcreteDataType

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change to list array

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify file_stream::create_stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify FileRegion

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* type alias

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove staled test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-11-24 12:41:54 +00:00
Weny XuandYingwen 47937961f6 feat(metric)!: enable sparse primary key encoding by default (#7195)
* feat(metric): enable sparse primary key encoding by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* Update src/mito-codec/src/key_values.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* feat: only allow setting primary key encoding for metric engine

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support deleting rows from logical region instead of physical region

This keeps the behavior the same as put. It's easier to support sparse
encoding for deleting logical regions. Now the metric engine doesn't
support delete rows from physical region directly.

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update sqlness

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2025-11-11 06:33:51 +00:00
shuiyisong a20ac4f9e5 feat: prefix option for timestamp index and value column (#7125)
* refactor: use GREPTIME_TIMESTAMP const

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: add config for default ts col name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: replace GREPTIME_TIMESTAMP with function get

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update config doc

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove opts on flownode and metasrv

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add validation for ts column name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use get_or_init to avoid test error

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fmt

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: using empty string to disable prefix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update comment

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: address CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2025-10-27 08:00:03 +00:00
dennis zhuangandCopilot 8a2371a05c feat: supports large string (#7097)
* feat: supports large string

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: add doc for extract_string_vector_values

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: refactor by cr comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: changes by cr comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: extract_string_vector_values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: remove large string type and refactor string vector

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: revert some changes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: adds large string type

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: impl default for StringSizeType

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: tests and test compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update sqlness tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove panic

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-17 01:46:11 +00:00
Ning Sun 2e6ea1167f refactor: update valueref coerce function name based on its semantics (#7098) 2025-10-16 09:11:40 +00:00
LFCandNing Sun 8fe17d43d5 chore: update rust to nightly 2025-10-01 (#7069)
* chore: update rust to nightly 2025-10-01

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: nix update

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-10-11 07:30:52 +00:00
Ning Sun 749a5ab165 feat: struct value and vector (#7033)
* feat: struct value

Signed-off-by: Ning Sun <sunning@greptime.com>

* feat: update for proto module

* feat: wip struct type

* feat: implement more vector operations

* feat: make datatype and api

* feat: reoslve some compilation issues

* feat: resolve all compilation issues

* chore: format update

* test: resolve tests

* test: test and refactor value-to-pb

* feat: add more tests and fix for value types

* chore: remove dbg

* feat: test and fix iterator

* fix: resolve struct_type issue

* refactor: use vec for struct items

* chore: update proto to main branch

* refactor: address some of review issues

* refactor: update for further review

* Add validation on new methods

* feat: update struct/list json serialization

* refactor: reimplement get in struct_vector

* refactor: struct vector functions

* refactor: fix lint issue

* refactor: address review comments

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2025-10-10 21:49:51 +00:00
Lei, HUANG c92ab4217f fix: avoid truncating SST statistics during flush (#6977)
fix/disable-parquet-stats-truncate:
 - **Update `memcomparable` Dependency**: Switched from crates.io to a Git repository for `memcomparable` in `Cargo.lock`, `mito-codec/Cargo.toml`, and removed it from `mito2/Cargo.toml`.
 - **Enhance Parquet Writer Properties**: Added `set_statistics_truncate_length` and `set_column_index_truncate_length` to `WriterProperties` in `parquet.rs`, `bulk/part.rs`, `partition_tree/data.rs`, and `writer.rs`.
 - **Add Test for Corrupt Scan**: Introduced a new test module `scan_corrupt.rs` in `mito2/src/engine` to verify handling of corrupt data.
 - **Update Test Data**: Modified test data in `flush.rs` to reflect changes in file sizes and sequences.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2025-09-17 03:02:52 +00:00
Ruihang Xia c9377e7c5a build: bump rust edition to 2024 (#6920)
* bump edition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* gen keyword

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime and env var

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more gen fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime of temporaries in tail expressions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy nested if

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy let and return

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-09-08 02:37:18 +00:00
Yingwen 8fc42aeb27 feat: Update parquet writer and indexer to support the flat format (#6866)
* feat: implements method to write flat batch for ParquetWriter

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add update method for flat RecordBatch in Indexer

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: calls indexer to write flat batch in ParquetWriter

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: handle empty projection for flat format

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: eval array in precise_filter_flat

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: cache column lookup result in inverted indexer

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add test

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support dict type in dense codec

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove read part in test as it need modifying the reader

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support dictionary type in other methods for dense codec

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: fulltext use string array directly

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-09-02 12:48:34 +00:00
Lei, HUANG dd3432e6ca chore: change encode raw values signature (#6869)
* chore/change-encode-raw-values-sig:
 ### Update Sparse Encoding to Use Byte Slices

 - **`bench_sparse_encoding.rs`**: Modified the `encode_raw_tag_value` function to use byte slices instead of `Bytes` for tag values.
 - **`sparse.rs`**: Updated the `encode_raw_tag_value` method in `SparsePrimaryKeyCodec` to accept byte slices (`&[u8]`) instead of `Bytes`. Adjusted related test cases to reflect this change.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/change-encode-raw-values-sig:
 ### Add `Clear` Trait Implementation for Byte Slices

 - Implemented the `Clear` trait for byte slices (`&[u8]`) in `repeated_field.rs` to enhance trait coverage and provide a default clear operation for byte slice types.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2025-09-01 09:53:08 +00:00
Lei, HUANGandYingwen d99734b97b perf: sparse encoder (#6809)
* perf/sparse-encoder:
 - **Update Dependencies**: Updated `criterion-plot` to version `0.5.0` and added `criterion` version `0.7.0` in `Cargo.lock`. Added `bytes` to `Cargo.toml` in `src/metric-engine`.
 - **Benchmarking**: Added a new benchmark for sparse encoding in `bench_sparse_encoding.rs` and updated `Cargo.toml` in `src/mito-codec` to include `criterion` as a dev-dependency.
 - **Sparse Encoding Enhancements**: Modified `SparsePrimaryKeyCodec` in `sparse.rs` to include new methods `encode_raw_tag_value` and `encode_internal`. Added public constants `RESERVED_COLUMN_ID_TSID` and `RESERVED_COLUMN_ID_TABLE_ID`.
 - **HTTP Server**: Made `try_decompress` function public in `prom_store.rs`.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* perf/sparse-encoder:
 Improve buffer handling in `sparse.rs`

 - Refactored buffer reservation logic to use `value_len` for clarity.
 - Optimized chunk processing by calculating `num_chunks` and `remainder` for efficient data handling.
 - Enhanced manual serialization of bytes to avoid byte-by-byte operations, improving performance.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* Update src/mito-codec/src/row_converter/sparse.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2025-08-26 04:10:11 +00:00
Yingwen d5575d3fa4 feat: add FlatConvertFormat to convert record batches in old format to the flat format (#6786)
* feat: add convert format to FlatReadFormat

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: test convert format

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: only convert string pks to dictionary

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-08-25 06:47:06 +00:00
LFC f9d2a89a0c chore: update datafusion family (#6675)
* chore: update datafusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* use official otel-arrow-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* rebase

Signed-off-by: luofucong <luofc@foxmail.com>

* use the official orc-rust

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* remove the empty lines

Signed-off-by: luofucong <luofc@foxmail.com>

* try following PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-08-15 12:41:49 +00:00
discord9 f07b1daed4 feat: struct vector (#6595)
* feat: struct vector

Signed-off-by: discord9 <discord9@163.com>

* fix: array2vector&arrow type2concrete type

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* chore: resolve some todos

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2025-07-29 08:22:27 +00:00
Yingwen eaf1e1198f refactor: Extract mito codec part into a new crate (#6307)
* chore: add a new crate mito-codec

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: port necessary mods for primary key codec

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: use codec utils in mito-codec

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove unused mods

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove Partition::is_partition_column()

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove duplicated test utils

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove unused comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: fix is_partition_column check

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2025-06-13 07:14:29 +00:00