Files
greptimedb/tests/compatibility/wire/raw_series_label_decode
discord9 7309800aef feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema

A raw series frame written with key-encoded labels carries the primary key
as opaque bytes; only the binding held by the scanner says how to turn
those bytes back into a metric name and labels. A consumer that receives
the frame without that binding cannot decode it, so the schema now
describes its own label layout.

The description is compact JSON under `greptime.series.label_decode`,
carrying a version, the metric name, the primary-key encoding, the ordered
key columns with their physical types, and the logical tags to expose
-- exactly what decoding reads. It is derived from the binding, so there
is one writer, and a reader that does not implement the version it finds
rejects the frame instead of guessing.

`unpack_raw_series` keeps its binding argument and now requires it to
agree with the description the schema declares, so the two cannot drift.
`PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key
layout, so the metadata-backed and description-backed paths share their
validation instead of growing a second decoder.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datatypes): compare label descriptions by what their order means

The frame's declared label-decode description was compared to the binding's with
plain equality, but the description has one part whose order carries no meaning:
the decoder sorts logical tags by name when it binds them, so the same tags listed
in another order denote the same labels and were being refused.

`same_label_layout` canonicalises exactly that part and leaves `primary_key` in
place, because for a dense key the array position is the encoding position: the same
columns in another order are a different key, not the same key reordered. Both
fields now say which order they mean.

Three checks go with it. The binding stays for a stated reason -- it establishes
identity, the physical region key layout a decoded identity came from -- rather than
because a count of call sites kept it. The post-pack schema comparison in the series
scan could not fire: the leaf already re-derives that schema and compares it, and
every `SeriesSchema` field is covered by the checks between them. And two test
groups re-asserted what the tracked v1_metadata fixture pins case by case.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(promql): decode a raw frame from the frame alone

`unpack_raw_series` took the label mode, the ordering and the caller's binding
along with the stream, so a consumer that received a frame from a peer still
needed a local copy of the binding to read it — the thing the frame's own
description exists to avoid, and the reason the description was freezing a wire
format for nobody.

The stream is now the only input: the field schema, the label mode, the label
decode description and the producer ordering all come from the frame's schema,
and every frame is validated whether or not a caller asked for it. `expected`
stays as an optional check on the schema that is declared — field names and their
order, and no evaluation grid — and supplies nothing to the decoder, so a caller
cannot smuggle in a layout the frame does not declare.

Three call sites compared the caller's binding against the frame and all three
were load-bearing: a peer's response is not trusted to be the region that was
planned, and a native plan's child is an arbitrary execution plan. Those became
`ensure_frame_matches_label_source`, which states what it compares: the label
layout, not the region. Two regions with the same tag ids and names compare equal,
and the doc now says so instead of claiming to establish identity.

A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers
`label_decode_spec()` instead; every consumer that decides whether to decode reads
the description, so the placeholder never stands in for "these labels are
complete".

The merge contract is the description, not the binding: inputs are compared by
the layout they declare, and a binding that could not be expressed on the wire — a
`RegionMetadata` snapshot — no longer makes the merge stricter than the frames it
merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a
copy, because the caller's binding keeps its own order and the table scan compares
producer tag lists positionally, and readers compare by equality.

The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to
cover. Rather than mirror all 25 datatype variants into a type this module owns, a
frozen fixture records the text of every variant and an exhaustive match makes a
new variant fail to compile until it is recorded. The version policy is written
next to the constant: exact-text comparison across nodes, unknown versions
rejected, any same-version change breaking — with the planning-time fallback
stated as intended and marked not implemented.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(store-api): make every key-encoded raw stream declare its layout

A raw series stream whose chunks carry key-encoded labels had to be decoded
against a scan-bound `RawLabelSource`, and two consumers compensated for
producers that stayed silent by deriving a description from that binding:
`pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels`
did not compensate, so an unframed producer stream reached it with no
description and a chunk carrying no labels, and failed.

The stream itself is the authority for its chunks. A key-encoded producer
knows its own region metadata and can state the layout it writes, exactly as a
frame states the layout it carries, so the split contract — "a producer
declares its identity, a frame declares its layout" — is replaced by one rule:
a stream whose chunks carry key-encoded labels declares the description, and a
stream whose chunks carry complete labels answers `None`.

mito2's `RawStream` caches the derivation at construction, so the declaration
exists before the first poll and for an empty stream. The trait documentation
states the rule and no longer exempts unframed producers. Both fallbacks are
gone: `pack_raw_series` reads the stream's declaration only, and rejects a
key-encoded stream that declares none rather than packing it as an
explicit-label frame, which would describe chunks carrying no labels as if they
carried all of them. The merge reads the declaration only.

Test fixtures that model key-encoded producers now declare their layout too;
fixtures that carry complete labels keep answering `None`. The new mito2 test
decodes a real produced key through the declared description for both dense and
sparse encodings, and checks the declaration is present before polling even for
an empty stream.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(common-query): let the declaration decide the frame mode alone

`pack_raw_series` rejected a key-encoded stream that declared no description
outright, so a stream that stays silent could not be packed at all. That check
was redundant: the declaration already decides the frame's label mode, and the
first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor`
with "raw chunk label mode does not match frame mode", so no such stream can
pack data either way.

What the check added was an early error for a producer that already breaks the
contract, in exchange for reading the stream's scan binding here — in the one
function whose rule is that the declaration is the only decode input. Drop it,
so pack keeps deciding the mode from the declaration and nothing else.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(store-api): assert the fixture's declared label decode description

The mock prepared source derives a `LabelDecodeSpec` from its label source and
reports it through `RawSeriesStream::label_decode_spec`, but no assertion read
it, so the derivation could drift from the chunks the fixture actually builds.
Check it against the description the fixture's own key columns imply: an
explicit source declares nothing, a primary-key source declares the metric, the
dense encoding, the one key column and its tag.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): let the frame's declaration be the only label authority

Two things still treated a caller-supplied binding as authoritative after the
declaration became the decode input:

- A merged stream forwarded its first input's scan-bound identity, though a
  merge is not a region scan and decodes from the declaration alone. It now
  reports no bound identity.
- `KeyColumn.tag` repeated an invariant every key column is a tag. The check
  moves to the one place a decoder is bound, so a snapshot built without
  validation is still rejected instead of silently decoded.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-21 19:11:24 +08:00
..

Raw-series label decode description compatibility

v1_metadata.json is a hand-authored, frozen set of Arrow schema metadata maps for the v1 raw-series frame. It records the established layout/content/labels/ time metadata plus the new greptime.series.label_decode entry, whose value is a JSON label decode description with an explicit version. The Rust unit test parses these literal maps through raw_series_frame_label_decode; the fixture is not produced by the schema builder. v1_physical_types.json, described under Physical types, is the one generated fixture here: it records the serde text of every ConcreteDataType variant.

Run the focused check:

cargo nextest run -p datatypes -E 'test(raw_series_label_decode_metadata_fixture)'

The fixture proves that a key-encoded frame's description decodes for both primary-key encodings, that the ordered key columns and logical tags survive the wire form with a mixed physical type set, that a logical tag must name a listed key column, and that the reader rejects a key frame without a description, an explicit frame that declares one, an unknown version, a missing or unknown field, a wrong field type, and a non-JSON value.

A timestamp-typed tag column has its own case in the crate's unit tests rather than here, so this fixture stays a single fixed layout.

The fixture does not prove interoperability with an old binary, nor does it test an end-to-end client or service. Separate pack and IPC tests check that a packed stream declares the description derived from its binding and that it agrees with the schema a consumer receives.

greptime.series.* metadata is compared as exact text by remote merges, so the text stored here is what the single writer produces: compact JSON in declaration order. A reader must never regenerate it differently; semantically equal text with different whitespace is a mismatch, not an accepted variant.

The description is versioned independently of the surrounding frame layout: a reader rejects any version it does not implement instead of guessing, so a future revision must add a new fixture case rather than relax this one.

Physical types

physical_type is ConcreteDataType's own serde form, so this description's text embeds the serialization of a type defined in src/datatypes/src/data_type.rs and its types/ modules. That shape is not restated or owned here: whatever ConcreteDataType's serde writes is what a description carries, and the two spellings that appear below — {"String":{"size_type":"Utf8"}} and {"UInt32":{}} — are examples of it rather than a vocabulary this fixture defines.

What freezes that shape is v1_physical_types.json: one entry per ConcreteDataType variant, {"variant": "<name>", "text": "<serde_json>"}, with text generated from serde_json::to_string rather than written by hand. The crate's every_datatype_round_trips_through_its_frozen_text test fails when a variant's serde no longer parses back or no longer re-serializes to the frozen text, and the_frozen_text_covers_every_datatype_variant fails when a variant of ConcreteDataType has no entry, because its exhaustive match names every variant. Changing ConcreteDataType's serde, or any text recorded here, changes the text two nodes compare, so it requires bumping LABEL_DECODE_VERSION and regenerating this fixture together.

logical_tags is written in the writer's canonical order (sorted by (column_id, name)). The order carries no meaning — the decoder sorts tags by name when it binds them — but fixing it means two producers of the same labels write the same bytes, which is what lets the same equality rule serve both a local comparison and the byte-exact remote one. The writer canonicalizes a copy of the description for exactly that reason, leaving the caller's binding order alone.

Run the physical-type check too:

cargo nextest run -p datatypes -E 'test(datatype) or test(frozen_text)'