feat(datatypes): describe label decoding in the raw series frame schema (#9786)

* feat(datatypes): describe label decoding in the raw series frame schema

A raw series frame written with key-encoded labels carries the primary key
as opaque bytes; only the binding held by the scanner says how to turn
those bytes back into a metric name and labels. A consumer that receives
the frame without that binding cannot decode it, so the schema now
describes its own label layout.

The description is compact JSON under `greptime.series.label_decode`,
carrying a version, the metric name, the primary-key encoding, the ordered
key columns with their physical types, and the logical tags to expose
-- exactly what decoding reads. It is derived from the binding, so there
is one writer, and a reader that does not implement the version it finds
rejects the frame instead of guessing.

`unpack_raw_series` keeps its binding argument and now requires it to
agree with the description the schema declares, so the two cannot drift.
`PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key
layout, so the metadata-backed and description-backed paths share their
validation instead of growing a second decoder.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datatypes): compare label descriptions by what their order means

The frame's declared label-decode description was compared to the binding's with
plain equality, but the description has one part whose order carries no meaning:
the decoder sorts logical tags by name when it binds them, so the same tags listed
in another order denote the same labels and were being refused.

`same_label_layout` canonicalises exactly that part and leaves `primary_key` in
place, because for a dense key the array position is the encoding position: the same
columns in another order are a different key, not the same key reordered. Both
fields now say which order they mean.

Three checks go with it. The binding stays for a stated reason -- it establishes
identity, the physical region key layout a decoded identity came from -- rather than
because a count of call sites kept it. The post-pack schema comparison in the series
scan could not fire: the leaf already re-derives that schema and compares it, and
every `SeriesSchema` field is covered by the checks between them. And two test
groups re-asserted what the tracked v1_metadata fixture pins case by case.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(promql): decode a raw frame from the frame alone

`unpack_raw_series` took the label mode, the ordering and the caller's binding
along with the stream, so a consumer that received a frame from a peer still
needed a local copy of the binding to read it — the thing the frame's own
description exists to avoid, and the reason the description was freezing a wire
format for nobody.

The stream is now the only input: the field schema, the label mode, the label
decode description and the producer ordering all come from the frame's schema,
and every frame is validated whether or not a caller asked for it. `expected`
stays as an optional check on the schema that is declared — field names and their
order, and no evaluation grid — and supplies nothing to the decoder, so a caller
cannot smuggle in a layout the frame does not declare.

Three call sites compared the caller's binding against the frame and all three
were load-bearing: a peer's response is not trusted to be the region that was
planned, and a native plan's child is an arbitrary execution plan. Those became
`ensure_frame_matches_label_source`, which states what it compares: the label
layout, not the region. Two regions with the same tag ids and names compare equal,
and the doc now says so instead of claiming to establish identity.

A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers
`label_decode_spec()` instead; every consumer that decides whether to decode reads
the description, so the placeholder never stands in for "these labels are
complete".

The merge contract is the description, not the binding: inputs are compared by
the layout they declare, and a binding that could not be expressed on the wire — a
`RegionMetadata` snapshot — no longer makes the merge stricter than the frames it
merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a
copy, because the caller's binding keeps its own order and the table scan compares
producer tag lists positionally, and readers compare by equality.

The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to
cover. Rather than mirror all 25 datatype variants into a type this module owns, a
frozen fixture records the text of every variant and an exhaustive match makes a
new variant fail to compile until it is recorded. The version policy is written
next to the constant: exact-text comparison across nodes, unknown versions
rejected, any same-version change breaking — with the planning-time fallback
stated as intended and marked not implemented.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(store-api): make every key-encoded raw stream declare its layout

A raw series stream whose chunks carry key-encoded labels had to be decoded
against a scan-bound `RawLabelSource`, and two consumers compensated for
producers that stayed silent by deriving a description from that binding:
`pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels`
did not compensate, so an unframed producer stream reached it with no
description and a chunk carrying no labels, and failed.

The stream itself is the authority for its chunks. A key-encoded producer
knows its own region metadata and can state the layout it writes, exactly as a
frame states the layout it carries, so the split contract — "a producer
declares its identity, a frame declares its layout" — is replaced by one rule:
a stream whose chunks carry key-encoded labels declares the description, and a
stream whose chunks carry complete labels answers `None`.

mito2's `RawStream` caches the derivation at construction, so the declaration
exists before the first poll and for an empty stream. The trait documentation
states the rule and no longer exempts unframed producers. Both fallbacks are
gone: `pack_raw_series` reads the stream's declaration only, and rejects a
key-encoded stream that declares none rather than packing it as an
explicit-label frame, which would describe chunks carrying no labels as if they
carried all of them. The merge reads the declaration only.

Test fixtures that model key-encoded producers now declare their layout too;
fixtures that carry complete labels keep answering `None`. The new mito2 test
decodes a real produced key through the declared description for both dense and
sparse encodings, and checks the declaration is present before polling even for
an empty stream.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(common-query): let the declaration decide the frame mode alone

`pack_raw_series` rejected a key-encoded stream that declared no description
outright, so a stream that stays silent could not be packed at all. That check
was redundant: the declaration already decides the frame's label mode, and the
first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor`
with "raw chunk label mode does not match frame mode", so no such stream can
pack data either way.

What the check added was an early error for a producer that already breaks the
contract, in exchange for reading the stream's scan binding here — in the one
function whose rule is that the declaration is the only decode input. Drop it,
so pack keeps deciding the mode from the declaration and nothing else.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(store-api): assert the fixture's declared label decode description

The mock prepared source derives a `LabelDecodeSpec` from its label source and
reports it through `RawSeriesStream::label_decode_spec`, but no assertion read
it, so the derivation could drift from the chunks the fixture actually builds.
Check it against the description the fixture's own key columns imply: an
explicit source declares nothing, a primary-key source declares the metric, the
dense encoding, the one key column and its tag.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): let the frame's declaration be the only label authority

Two things still treated a caller-supplied binding as authoritative after the
declaration became the decode input:

- A merged stream forwarded its first input's scan-bound identity, though a
  merge is not a region scan and decodes from the declaration alone. It now
  reports no bound identity.
- `KeyColumn.tag` repeated an invariant every key column is a tag. The check
  moves to the one place a decoder is bound, so a snapshot built without
  validation is still rejected instead of silently decoded.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
This commit is contained in:
discord9
2026-10-21 19:11:24 +08:00
committed by GitHub
parent f235434b7f
commit 7309800aef
25 changed files with 3265 additions and 602 deletions
+327 -3
View File
@@ -18,11 +18,45 @@
//! chunk and [`RawSeriesFrameBuilder`] emits rows directly as Arrow arrays once a row or estimated
//! byte bound is reached. Decoding reuses one prepared schema per stream through
//! [`RawSeriesFrameDecoder`]. Column layout, metadata, and wire strings stay on the frozen `v1`
//! contract.
//! contract; a key-encoded frame additionally declares a versioned [`LabelDecodeSpec`] so its
//! labels can be decoded from the received schema alone.
//!
//! # Label decode description versioning
//!
//! The description is one `greptime.series.label_decode` metadata entry, and remote plans compare
//! `greptime.series.*` metadata **as exact text, byte for byte**: a frontend and a datanode that
//! write different text for the same labels disagree about the schema, and the query fails instead
//! of falling back. The rules that follow from that are:
//!
//! - A reader rejects a version it does not implement, rather than guessing. This module writes and
//! reads [`LABEL_DECODE_VERSION`] only.
//! - [`LABEL_DECODE_VERSION`] covers the whole text, including each key column's
//! [`LabelDecodeColumn::physical_type`]. That field is a [`ConcreteDataType`], so the text embeds
//! [`ConcreteDataType`]'s serde shape: it is not spelled here, and a change in that type's
//! serialization does alter this text. What keeps such a change from going unnoticed is the
//! frozen fixture `tests/compatibility/wire/raw_series_label_decode/v1_physical_types.json`,
//! which records the text of every datatype variant and fails the crate's tests when any of them
//! is serialized differently.
//! - Any change to the shape of this text — a renamed or added field, a different
//! [`ConcreteDataType`] spelling, a different `logical_tags` order — is a **breaking** change at
//! the same version, because it makes two nodes of different builds disagree. Such a change must
//! bump [`LABEL_DECODE_VERSION`] and update that fixture with it.
//! - The text is canonical: [`LabelDecodeSpec::canonicalize`] fixes the only part whose order
//! carries no meaning, so equal labels always write equal bytes.
//!
//! Deployment rule: during a rolling upgrade a frame may carry a version the receiving node does not
//! implement, and a byte-for-byte schema comparison may fail for that reason alone. The intended
//! behaviour is that such a mismatch disables the native series path at planning time and falls back
//! to the generic path, instead of failing the query at execution.
//!
//! TODO(narthex): implement that planning-time fallback. No planner reads
//! `raw_series_frame_label_decode` to decide support today, so a version or text mismatch currently
//! surfaces as an execution-time error from the raw-frame decode or the remote schema comparison.
#[cfg(test)]
mod tests;
use std::collections::HashSet;
use std::sync::Arc;
use arrow::array::{
@@ -33,12 +67,14 @@ use arrow::buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
use arrow::compute::concat;
use arrow::datatypes::{DataType, Field, Fields, Schema, SchemaRef};
use arrow::record_batch::{RecordBatch, RecordBatchOptions};
use serde::{Deserialize, Serialize};
use crate::data_type::ConcreteDataType;
use crate::error::{InvalidSeriesSnafu, Result};
use crate::series::histogram::{HistogramBatch, fields as histogram_fields};
use crate::series::raw::{RawSeriesChunk, RawTimestamps};
use crate::series::samples::{
DenseSamplesRef, HistogramSamples, Samples, SeriesFields, SeriesSchema,
DenseSamplesRef, FieldSpec, HistogramSamples, Samples, SeriesFields, SeriesSchema,
};
use crate::series::{Label, MetricName, invalid_series};
@@ -46,6 +82,12 @@ const LAYOUT: &str = "greptime.series.layout";
const CONTENT: &str = "greptime.series.content";
const LABELS: &str = "greptime.series.labels";
const TIME: &str = "greptime.series.time";
const GRID: &str = "greptime.series.grid";
/// Metadata key of the versioned label decode description carried by key-encoded frames.
const LABEL_DECODE: &str = "greptime.series.label_decode";
/// Version of the label decode description this revision writes and reads.
pub const LABEL_DECODE_VERSION: u32 = 1;
/// Whether a raw frame carries only the complete key or explicit metric labels as well.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -56,7 +98,166 @@ pub enum RawLabelMode {
Explicit,
}
/// Primary-key encoding of the labels a key-encoded frame carries.
///
/// This mirrors the storage contract's `PrimaryKeyEncoding` without making this crate depend on
/// it; Dense and Sparse keys use different decode layouts.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum LabelDecodeEncoding {
/// Dense key: every primary-key field is encoded in order, with no ids on the wire.
Dense,
/// Sparse key: `(column id, value)` pairs, so the layout only frames the field stream.
Sparse,
}
/// One ordered key column needed to decode key-encoded labels.
///
/// Only tag columns reach this list: region metadata rejects a primary key that holds the time
/// index or a non-`Tag` column, and it requires the primary key to name every tag column. The
/// array's position is therefore the encoding position, with nothing to disambiguate.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct LabelDecodeColumn {
/// Physical column id of this key column.
pub column_id: u32,
/// Physical column name; a logical tag of the same id must repeat it.
pub name: String,
/// Physical data type of the encoded value, in [`ConcreteDataType`]'s own serde form. Dense
/// keys carry no types on the wire, so the codec is reconstructed from this exact ordered type
/// list.
pub physical_type: ConcreteDataType,
}
/// One logical tag name bound to a primary-key column.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct LabelDecodeTag {
/// Physical column id this logical tag reads.
pub column_id: u32,
/// Logical tag name exposed to queries.
pub name: String,
}
/// Versioned, self-contained description of how to decode key-encoded labels.
///
/// It carries exactly what decoding a primary-key label source reads and nothing else: no region
/// identity, no field columns, no time index. A consumer that receives the frame schema can
/// therefore reconstruct an equivalent decoder without a scan-side binding, while a description
/// that is missing, of an unknown version, or structurally inconsistent is a hard error instead of
/// a silent fallback.
///
/// It is JSON-encoded in one schema metadata entry, always through
/// [`raw_series_frame_schema_with_label_decode`], so both the producing and the expecting side of a
/// `greptime.series.*` exact metadata comparison see byte-identical text. The text is
/// self-describing but not a general interchange format: readers reject unknown fields here only
/// at this structure's own levels, while [`LabelDecodeColumn::physical_type`] is decoded by
/// [`ConcreteDataType`]'s own representation and is therefore exactly as stable as that type's
/// serde. The frozen text this revision writes is pinned by
/// `tests/compatibility/wire/raw_series_label_decode/v1_physical_types.json`; see the module
/// documentation for what a change to it means.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct LabelDecodeSpec {
/// Description version; see [`LABEL_DECODE_VERSION`].
pub version: u32,
/// Metric name attached to every decoded label set; may be empty.
pub metric_name: String,
/// Primary-key encoding that determines the decode layout.
pub encoding: LabelDecodeEncoding,
/// Every tag column of the primary key, in primary-key encoding order. This is exactly the
/// set of encoded key fields: region metadata forbids a primary key from naming the time
/// index or a field column, and requires it to name every tag column, so no time-index or
/// field entry can appear here. The order is significant and is compared as written: a dense
/// decode reads field *n* at position *n*, so the same columns in another order describe a
/// different key, not the same key in another order.
pub primary_key: Vec<LabelDecodeColumn>,
/// Logical tags, as bound by the producer, in [`LabelDecodeSpec::canonicalize`] order.
///
/// The order carries no meaning: the decoder sorts the tags by name when it binds them
/// (`src/mito-codec/src/labels.rs:235`), so a decoded label set does not depend on it. It is
/// nevertheless fixed rather than left to the caller, because this description is compared as
/// exact text across nodes: two producers with the same tags in another order must write
/// byte-identical text, or one side rejects the other as a schema mismatch.
///
/// The semantic checks on these tags (reserved and internal names, name equality with the bound
/// column, membership in `primary_key`) are the decoder's, matching what a typed binding gets.
pub logical_tags: Vec<LabelDecodeTag>,
}
impl LabelDecodeSpec {
/// Puts the order-insensitive part of the description in its canonical order.
///
/// [`raw_series_frame_schema_with_label_decode`] applies this to a copy before serializing, and
/// so does the description a binding derives (in `common_query::series_frame`), so both sides of
/// a text comparison produce the same bytes and a reader can compare descriptions structurally.
/// `primary_key` is deliberately untouched: its order is the encoding order.
///
/// The writer canonicalizes a **copy** on purpose. A caller's binding keeps its own
/// `logical_tags` order, and that order is meaningful elsewhere: `src/table/src/table/scan.rs`
/// compares two bindings' tag lists positionally (`lt.iter().zip(rt.iter())`), so reordering the
/// caller's list here would make two bindings for the same labels compare unequal. Only the
/// serialized text is canonical; nothing outside it is rewritten.
///
/// Sorting by `(column_id, name)` is a total order for a validated description, which names
/// each tag column and each tag name at most once, so the canonical form is unique.
pub fn canonicalize(&mut self) {
self.logical_tags
.sort_unstable_by(|a, b| (a.column_id, &a.name).cmp(&(b.column_id, &b.name)));
}
/// Checks the version and the structural invariants a decoder relies on.
///
/// Semantic binding checks (reserved names, internal columns, name equality against the bound
/// column) stay in the decoder so both a typed binding and a decoded description fail with the
/// same errors.
pub fn validate(&self) -> Result<()> {
if self.version != LABEL_DECODE_VERSION {
return Err(err(format!(
"unsupported label decode description version {}",
self.version
)));
}
let mut column_ids = HashSet::with_capacity(self.primary_key.len());
for column in &self.primary_key {
if !column_ids.insert(column.column_id) {
return Err(err(format!(
"duplicate primary-key column id {} in label decode description",
column.column_id
)));
}
}
let mut tag_ids = HashSet::with_capacity(self.logical_tags.len());
let mut tag_names = HashSet::with_capacity(self.logical_tags.len());
for tag in &self.logical_tags {
if !tag_ids.insert(tag.column_id) {
return Err(err(format!(
"duplicate logical tag column id {} in label decode description",
tag.column_id
)));
}
if !tag_names.insert(tag.name.as_str()) {
return Err(err(format!(
"duplicate logical tag {} in label decode description",
tag.name
)));
}
if !column_ids.contains(&tag.column_id) {
return Err(err(format!(
"logical tag {} is not a primary-key column of the label decode description",
tag.name
)));
}
}
Ok(())
}
}
/// Builds the typed schema for a raw frame, preserving logical field names and order.
///
/// This is the column layout shared with the frame builder and decoder; it deliberately carries no
/// label decode description. A key-encoded stream must be declared through
/// [`raw_series_frame_schema_with_label_decode`], which refuses such a layout without one.
pub fn raw_series_frame_schema(schema: &SeriesSchema, mode: RawLabelMode) -> Result<SchemaRef> {
if schema.eval_grid.is_some() {
return invalid_series("raw series schema cannot have an evaluation grid");
@@ -137,6 +338,129 @@ pub fn raw_series_frame_schema(schema: &SeriesSchema, mode: RawLabelMode) -> Res
Ok(Arc::new(Schema::new_with_metadata(columns, metadata)))
}
/// Builds the typed schema for a raw frame together with its label decode description.
///
/// The description is written only for [`RawLabelMode::Key`], where labels stay encoded in
/// `read_key`; explicit-label frames carry complete labels and must not declare one. Both
/// directions are enforced here so a frame cannot claim a label layout it does not describe.
pub fn raw_series_frame_schema_with_label_decode(
schema: &SeriesSchema,
mode: RawLabelMode,
decode: Option<&LabelDecodeSpec>,
) -> Result<SchemaRef> {
match (mode, decode) {
(RawLabelMode::Key, None) => {
return Err(err(
"key-encoded raw frames must carry a label decode description",
));
}
(RawLabelMode::Explicit, Some(_)) => {
return Err(err("explicit raw frames carry no label decode description"));
}
(RawLabelMode::Key, Some(_)) | (RawLabelMode::Explicit, None) => {}
}
let layout = raw_series_frame_schema(schema, mode)?;
let Some(decode) = decode else {
return Ok(layout);
};
// Writing is faithful: the description records what the producer declares, and nothing is
// rewritten or dropped. The one exception is the order of `logical_tags`, which carries no
// meaning to a decoder and which this text nevertheless has to fix so two producers of the same
// labels write the same bytes; a copy is canonicalized and the caller's value is untouched. A
// description this revision cannot honour is rejected when it is read, which is the only
// direction a consumer can rely on.
let mut canonical = decode.clone();
canonical.canonicalize();
let encoded = serde_json::to_string(&canonical)
.map_err(|e| err(format!("cannot encode label decode description: {e}")))?;
let mut metadata = layout.metadata().clone();
metadata.insert(LABEL_DECODE.to_string(), encoded);
Ok(Arc::new(Schema::new_with_metadata(
layout.fields().clone(),
metadata,
)))
}
/// Recovers the label decode description a raw frame schema declares.
///
/// Returns `None` for an explicit-label frame, which must not declare one. A key-encoded frame must
/// declare one, so a missing description is an error rather than a reason to reuse a caller-side
/// binding. A present description is parsed strictly: an unknown version, a missing or unknown
/// field at this structure's own levels, a type mismatch, or a description whose logical tags do
/// not name listed key columns is an error, never a silent default. Only the description's own
/// levels reject unknown fields; [`LabelDecodeColumn::physical_type`] is a
/// [`ConcreteDataType`] and accepts whatever that type's own serde accepts.
pub fn raw_series_frame_label_decode(schema: &Schema) -> Result<Option<LabelDecodeSpec>> {
let mode = match schema.metadata().get(LABELS).map(String::as_str) {
Some("key") => RawLabelMode::Key,
Some("explicit") => RawLabelMode::Explicit,
_ => return Err(err("raw frame schema declares no known label mode")),
};
let decode = match schema.metadata().get(LABEL_DECODE) {
None => None,
Some(encoded) => {
let decode: LabelDecodeSpec = serde_json::from_str(encoded)
.map_err(|e| err(format!("malformed label decode description: {e}")))?;
decode.validate()?;
Some(decode)
}
};
match (mode, &decode) {
(RawLabelMode::Key, None) => Err(err(
"key-encoded raw frame declares no label decode description",
)),
(RawLabelMode::Explicit, Some(_)) => Err(err(
"explicit raw frame declares a label decode description",
)),
(RawLabelMode::Key, Some(_)) | (RawLabelMode::Explicit, None) => Ok(decode),
}
}
/// Reconstructs the series schema a raw frame declares.
///
/// A raw frame names its payload fields by name only: every member of the `fields` struct carries
/// the same payload type ([`raw_series_frame_schema`]), and [`SeriesSchema`] holds nothing else per
/// field. The reconstruction is therefore exact rather than a projection — the declared names in
/// declared order, and no evaluation grid, which a raw frame cannot carry.
///
/// The reserved layout, content, and time metadata are checked here together with the absence of an
/// evaluation grid, so a frame this revision cannot read is rejected instead of being silently
/// interpreted as the current layout.
pub fn raw_series_frame_series_schema(schema: &Schema) -> Result<SeriesSchema> {
for (key, expected) in [(LAYOUT, "v1"), (CONTENT, "raw"), (TIME, "timestamps")] {
if schema.metadata().get(key).map(String::as_str) != Some(expected) {
return Err(err(format!("raw frame declares no known {key} value")));
}
}
if schema
.metadata()
.keys()
.any(|key| key == GRID || key.starts_with("greptime.series.grid."))
{
return Err(err("raw series schema cannot have an evaluation grid"));
}
let fields = schema
.fields()
.iter()
.find(|field| field.name() == "fields")
.ok_or_else(|| err("raw frame declares no fields column"))?;
let DataType::Struct(members) = fields.data_type() else {
return Err(err("raw frame fields column is not a struct"));
};
if members.is_empty() {
return Err(err("raw series schema must have fields"));
}
Ok(SeriesSchema {
fields: members
.iter()
.map(|member| FieldSpec {
name: Arc::from(member.name().as_str()),
})
.collect(),
eval_grid: None,
})
}
/// Bounded multi-row builder for finite raw frame batches.
///
/// Rows accumulate until the row limit or the estimated byte limit is reached; the next
@@ -871,7 +1195,7 @@ fn decode_raw_series_frame(
if actual
.metadata()
.keys()
.any(|key| key == "greptime.series.grid" || key.starts_with("greptime.series.grid."))
.any(|key| key == GRID || key.starts_with("greptime.series.grid."))
|| actual.fields() != expected.fields()
{
return invalid_series("raw frame schema mismatch");
+527
View File
@@ -3058,3 +3058,530 @@ fn decoder_reuses_cached_schema_and_rejects_schema_evolution() {
assert!(explicit_decoder.decode(&first).is_err());
assert!(key_decoder.decode(&nameless).is_err());
}
/// A dense description carrying the ordered key columns and the logical tags the decoder reads.
fn dense_description() -> LabelDecodeSpec {
LabelDecodeSpec {
version: LABEL_DECODE_VERSION,
metric_name: "cpu_usage".to_string(),
encoding: LabelDecodeEncoding::Dense,
primary_key: vec![
LabelDecodeColumn {
column_id: 1,
name: "host".to_string(),
physical_type: crate::data_type::ConcreteDataType::string_datatype(),
},
LabelDecodeColumn {
column_id: 2,
name: "region".to_string(),
physical_type: crate::data_type::ConcreteDataType::string_datatype(),
},
],
logical_tags: vec![LabelDecodeTag {
column_id: 1,
name: "host".to_string(),
}],
}
}
#[test]
fn label_decode_description_roundtrips_through_schema_metadata() {
let description = dense_description();
let key = raw_series_frame_schema_with_label_decode(
&test_schema(&["value"]),
RawLabelMode::Key,
Some(&description),
)
.unwrap();
assert_eq!(
raw_series_frame_label_decode(&key).unwrap(),
Some(description.clone())
);
// The description is metadata-only: the column contract is untouched.
assert_eq!(
key.fields(),
raw_series_frame_schema(&test_schema(&["value"]), RawLabelMode::Key)
.unwrap()
.fields()
);
assert_eq!(key.metadata().get(LABELS).map(String::as_str), Some("key"));
}
#[test]
fn explicit_frames_declare_no_label_decode_description() {
let schema = test_schema(&["value"]);
let explicit =
raw_series_frame_schema_with_label_decode(&schema, RawLabelMode::Explicit, None).unwrap();
assert!(explicit.metadata().get(LABEL_DECODE).is_none());
assert_eq!(raw_series_frame_label_decode(&explicit).unwrap(), None);
// A key frame without a description, or an explicit frame with one, is rejected instead of
// being written as a frame the consumer cannot decode.
assert!(
raw_series_frame_schema_with_label_decode(&schema, RawLabelMode::Key, None).is_err(),
"key-encoded frames must describe their labels"
);
assert!(
raw_series_frame_schema_with_label_decode(
&schema,
RawLabelMode::Explicit,
Some(&dense_description())
)
.is_err(),
"explicit frames carry complete labels"
);
}
/// The raw frame's own columns with exactly the supplied metadata map.
fn schema_with_metadata(metadata: Vec<(String, String)>) -> Schema {
let layout = raw_series_frame_schema(&test_schema(&["value"]), RawLabelMode::Key).unwrap();
Schema::new_with_metadata(layout.fields().clone(), metadata.into_iter().collect())
}
/// The four metadata entries every raw frame declares, with the given label mode.
fn base_metadata(labels: &str) -> Vec<(String, String)> {
[
("greptime.series.layout", "v1"),
("greptime.series.content", "raw"),
("greptime.series.labels", labels),
("greptime.series.time", "timestamps"),
]
.into_iter()
.map(|(key, value)| (key.to_string(), value.to_string()))
.collect()
}
#[test]
fn label_decode_description_rejects_inconsistent_bindings() {
let schema = test_schema(&["value"]);
let mut cases = Vec::new();
let mut duplicate_column = dense_description();
duplicate_column.primary_key[1].column_id = 1;
cases.push(("duplicate primary-key column", duplicate_column));
let mut unbound_tag = dense_description();
unbound_tag.logical_tags[0].column_id = 3;
cases.push(("logical tag outside the primary key", unbound_tag));
let mut duplicate_tag = dense_description();
let tag = duplicate_tag.logical_tags[0].clone();
duplicate_tag.logical_tags.push(tag);
cases.push(("duplicate logical tag", duplicate_tag));
for (label, description) in cases {
// Writing is faithful: the description records what the producer declares. Reading is
// where a description this revision cannot honour is rejected, so a producer cannot turn a
// bad binding into a frame a consumer would decode under silently.
let frame = raw_series_frame_schema_with_label_decode(
&schema,
RawLabelMode::Key,
Some(&description),
)
.unwrap_or_else(|error| panic!("{label}: {error}"));
// The writer records what the producer declares, so the rejection has to come from the
// reader; one assertion covers it, since the frame it reads is the text just written.
assert!(
raw_series_frame_label_decode(&frame).is_err(),
"{label}: an inconsistent description is rejected when read"
);
}
}
/// A frame declaring no label mode has no label layout to describe. The rest of the parse and
/// validation rejections (unknown version, missing/unknown/wrongly typed fields, malformed JSON,
/// and mode and description disagreeing in either direction) are pinned case for case by
/// [`raw_series_label_decode_metadata_fixture`], so they are not re-asserted here.
#[test]
fn label_decode_rejects_a_frame_without_a_label_mode() {
let without_mode = base_metadata("key")
.into_iter()
.filter(|(key, _)| key != LABELS)
.collect();
assert!(
raw_series_frame_label_decode(&schema_with_metadata(without_mode)).is_err(),
"a frame without a label mode has no label layout to describe"
);
}
#[test]
fn raw_series_label_decode_metadata_fixture() {
use std::collections::HashMap;
#[derive(serde::Deserialize)]
struct Case {
name: String,
metadata: HashMap<String, String>,
/// The encoding a decoding consumer expects, or `null` when the fixture must be rejected.
encoding: Option<String>,
}
let cases: Vec<Case> = serde_json::from_str(include_str!(
"../../../../../tests/compatibility/wire/raw_series_label_decode/v1_metadata.json"
))
.unwrap();
for case in cases {
let layout = raw_series_frame_schema(&test_schema(&["value"]), RawLabelMode::Key).unwrap();
let metadata = case
.metadata
.iter()
.map(|(key, value)| (key.clone(), value.clone()))
.collect();
let frame_schema = Schema::new_with_metadata(layout.fields().clone(), metadata);
match case.encoding.as_deref() {
None => assert!(
raw_series_frame_label_decode(&frame_schema).is_err(),
"{}: must be rejected",
case.name
),
Some(expected) => {
let decode = raw_series_frame_label_decode(&frame_schema)
.unwrap_or_else(|error| panic!("{}: {error}", case.name))
.unwrap_or_else(|| panic!("{}: no description", case.name));
let encoding = match decode.encoding {
LabelDecodeEncoding::Dense => "dense",
LabelDecodeEncoding::Sparse => "sparse",
};
assert_eq!(encoding, expected, "{}", case.name);
assert_eq!(decode.version, LABEL_DECODE_VERSION, "{}", case.name);
assert_eq!(decode.metric_name, "cpu_usage", "{}", case.name);
assert_eq!(
decode.primary_key,
vec![
LabelDecodeColumn {
column_id: 1,
name: "host".to_string(),
physical_type: crate::data_type::ConcreteDataType::string_datatype(),
},
LabelDecodeColumn {
column_id: 2,
name: "shard".to_string(),
physical_type: crate::data_type::ConcreteDataType::uint32_datatype(),
},
],
"{}: the ordered key columns are decoded in place",
case.name
);
assert_eq!(
decode.logical_tags,
vec![
LabelDecodeTag {
column_id: 1,
name: "host".to_string(),
},
LabelDecodeTag {
column_id: 2,
name: "shard".to_string(),
},
],
"{}",
case.name
);
if expected == "dense" {
// The fixture text is the encoder's own output, byte for byte: re-encoding the
// description it decodes to must reproduce it exactly.
let roundtrip = raw_series_frame_schema_with_label_decode(
&test_schema(&["value"]),
RawLabelMode::Key,
Some(&decode),
)
.unwrap();
assert_eq!(
roundtrip.metadata().get(LABEL_DECODE),
case.metadata.get(LABEL_DECODE),
"{}: re-encoding reproduces the fixture text",
case.name
);
}
}
}
}
}
/// A tag column typed as a timestamp is described like any other key column: the description
/// carries its physical type, and the text is exactly what the encoder wrote.
#[test]
fn label_decode_description_carries_a_timestamp_typed_tag_column() {
let mut description = dense_description();
description.primary_key[1].name = "observed".to_string();
description.primary_key[1].physical_type =
crate::data_type::ConcreteDataType::timestamp_millisecond_datatype();
description.logical_tags.push(LabelDecodeTag {
column_id: 2,
name: "observed".to_string(),
});
let frame = raw_series_frame_schema_with_label_decode(
&test_schema(&["value"]),
RawLabelMode::Key,
Some(&description),
)
.unwrap();
let text = frame.metadata().get(LABEL_DECODE).unwrap();
// The expected text is the encoder's real output for a timestamp-typed tag column.
assert!(
text.contains("\"physical_type\":{\"Timestamp\":{\"Millisecond\":null}}"),
"the encoder wrote a timestamp type: {text}"
);
let recovered = raw_series_frame_label_decode(&frame).unwrap().unwrap();
assert_eq!(recovered, description);
assert_eq!(
recovered.primary_key[1].physical_type,
crate::data_type::ConcreteDataType::timestamp_millisecond_datatype()
);
}
/// Two producers that list the same logical tags in a different order write the same bytes.
///
/// The tag order carries no meaning to a decoder, so the writer sorts its own copy before
/// serializing and leaves the caller's value alone. The encoding order of `primary_key` is a
/// different thing entirely: it is the order of the key columns, so it is written as given.
#[test]
fn label_decode_text_is_independent_of_the_callers_tag_order() {
let mut ascending = dense_description();
ascending.logical_tags = vec![
LabelDecodeTag {
column_id: 1,
name: "host".to_string(),
},
LabelDecodeTag {
column_id: 2,
name: "region".to_string(),
},
];
let mut descending = ascending.clone();
descending.logical_tags.reverse();
// The encoding order of the key columns is not the tag order: reverse it in the second
// description so a writer that sorted `primary_key` too would be caught below.
descending.primary_key.reverse();
let write = |description: &LabelDecodeSpec| {
let frame = raw_series_frame_schema_with_label_decode(
&test_schema(&["value"]),
RawLabelMode::Key,
Some(description),
)
.unwrap();
frame.metadata().get(LABEL_DECODE).unwrap().clone()
};
let same_tags = {
let mut reordered = ascending.clone();
reordered.logical_tags.reverse();
write(&reordered)
};
assert_eq!(
write(&ascending),
same_tags,
"two tag orders wrote different bytes"
);
assert_eq!(
ascending.logical_tags,
vec![
LabelDecodeTag {
column_id: 1,
name: "host".to_string(),
},
LabelDecodeTag {
column_id: 2,
name: "region".to_string(),
},
],
"the writer reordered the caller's description"
);
// The key columns keep the order they were given, so the two descriptions are different
// frames rather than the same bytes: that is the part canonicalization must not touch.
assert_ne!(
write(&ascending),
write(&descending),
"the writer reordered the encoding columns"
);
let recovered = raw_series_frame_label_decode(
&raw_series_frame_schema_with_label_decode(
&test_schema(&["value"]),
RawLabelMode::Key,
Some(&descending),
)
.unwrap(),
)
.unwrap()
.unwrap();
assert_eq!(
recovered.primary_key, descending.primary_key,
"the recovered key order is not the encoding order"
);
}
/// One entry of `v1_physical_types.json`: a [`ConcreteDataType`] variant and the exact text this
/// revision's serde writes for a representative value of it.
#[derive(serde::Deserialize)]
struct FrozenPhysicalType {
/// The [`ConcreteDataType`] variant, as [`concrete_datatype_variants`] spells it.
variant: String,
/// The `serde_json` text this revision writes for that variant's representative value.
text: String,
}
/// Spells every [`ConcreteDataType`] variant exactly once, together with one representative value of
/// it.
///
/// Everything that has to notice a new variant is derived from this one list:
///
/// - [`variant_name`] matches on it with **no wildcard arm**, so adding a variant to
/// [`ConcreteDataType`] is a compile error until the variant is added here.
/// - [`datatype_representatives`] yields one value per listed variant, so the same edit is what
/// supplies the value the tests read the serde text from.
/// - [`VARIANT_NAMES`] is generated from it too, which is what
/// [`the_frozen_text_covers_every_datatype_variant`] requires a fixture entry for.
macro_rules! concrete_datatype_variants {
($($variant:ident => $value:expr),+ $(,)?) => {
/// The [`ConcreteDataType`] variant a value belongs to, spelled as the fixture names it.
///
/// Exhaustive on purpose: a new variant of [`ConcreteDataType`] makes this match fail to
/// compile, which is what makes a missing fixture entry impossible to land.
fn variant_name(data_type: &ConcreteDataType) -> &'static str {
match data_type {
$(ConcreteDataType::$variant(_) => stringify!($variant),)+
}
}
/// One representative value per variant, in the order listed above.
fn datatype_representatives() -> Vec<ConcreteDataType> {
vec![$($value),+]
}
/// Every [`ConcreteDataType`] variant name, in the order listed above.
const VARIANT_NAMES: &[&str] = &[$(stringify!($variant)),+];
};
}
concrete_datatype_variants!(
Null => ConcreteDataType::null_datatype(),
Boolean => ConcreteDataType::boolean_datatype(),
Int8 => ConcreteDataType::int8_datatype(),
Int16 => ConcreteDataType::int16_datatype(),
Int32 => ConcreteDataType::int32_datatype(),
Int64 => ConcreteDataType::int64_datatype(),
UInt8 => ConcreteDataType::uint8_datatype(),
UInt16 => ConcreteDataType::uint16_datatype(),
UInt32 => ConcreteDataType::uint32_datatype(),
UInt64 => ConcreteDataType::uint64_datatype(),
Float32 => ConcreteDataType::float32_datatype(),
Float64 => ConcreteDataType::float64_datatype(),
Decimal128 => ConcreteDataType::decimal128_datatype(10, 2),
Binary => ConcreteDataType::binary_datatype(),
String => ConcreteDataType::string_datatype(),
Date => ConcreteDataType::date_datatype(),
Timestamp => ConcreteDataType::timestamp_millisecond_datatype(),
Time => ConcreteDataType::time_microsecond_datatype(),
Duration => ConcreteDataType::duration_nanosecond_datatype(),
Interval => ConcreteDataType::interval_month_day_nano_datatype(),
List => ConcreteDataType::list_datatype(Arc::new(ConcreteDataType::string_datatype())),
Dictionary => ConcreteDataType::dictionary_datatype(
ConcreteDataType::int32_datatype(),
ConcreteDataType::string_datatype(),
),
Struct => ConcreteDataType::struct_datatype(crate::types::StructType::new(Arc::new(vec![
crate::types::StructField::new("host", ConcreteDataType::string_datatype(), false),
crate::types::StructField::new("port", ConcreteDataType::uint16_datatype(), true),
]))),
Json => ConcreteDataType::json_datatype(),
Vector => ConcreteDataType::vector_datatype(3),
);
/// The frozen text of every [`ConcreteDataType`] variant, as a label decode description embeds it.
const FROZEN_PHYSICAL_TYPES: &str = include_str!(
"../../../../../tests/compatibility/wire/raw_series_label_decode/v1_physical_types.json"
);
fn frozen_physical_types() -> Vec<FrozenPhysicalType> {
serde_json::from_str(FROZEN_PHYSICAL_TYPES).expect("the frozen physical-type fixture parses")
}
/// The representative value of one variant, named the way the fixture names it.
fn representative(variant: &str) -> ConcreteDataType {
VARIANT_NAMES
.iter()
.zip(datatype_representatives())
.find(|(name, _)| **name == variant)
.unwrap_or_else(|| panic!("{variant}: no representative value is listed for this variant"))
.1
}
/// The text a description embeds for a key column's type is not this module's to spell: it is
/// [`ConcreteDataType`]'s own serde, so it changes whenever that type's serde does.
///
/// `v1_physical_types.json` freezes that text for every variant. This test fails when a variant no
/// longer parses back from its frozen text, when it no longer re-serializes to it, or when the
/// parsed value is no longer the type that variant names — which is what a silent serde change
/// looks like. Since the frame text is compared byte for byte across nodes, such a change is a
/// [`LABEL_DECODE_VERSION`] bump, and this test is the alarm that raises it.
#[test]
fn every_datatype_round_trips_through_its_frozen_text() {
use crate::data_type::DataType as _;
for entry in frozen_physical_types() {
let parsed: ConcreteDataType = serde_json::from_str(&entry.text).unwrap_or_else(|error| {
panic!("{}: {} cannot be read: {error}", entry.variant, entry.text)
});
assert_eq!(
serde_json::to_string(&parsed).unwrap(),
entry.text,
"{}: re-serializes to the frozen text",
entry.variant
);
assert_eq!(
parsed.name(),
representative(&entry.variant).name(),
"{}: the frozen text still names the same type",
entry.variant
);
assert_eq!(
parsed,
representative(&entry.variant),
"{}: the frozen text still describes the same value",
entry.variant
);
}
}
/// The fixture above only freezes a variant if it has an entry, so every variant must have one.
///
/// This test is what makes that hold: [`variant_name`] matches every [`ConcreteDataType`] variant
/// with no wildcard arm, so a new variant does not compile until it is listed in
/// [`concrete_datatype_variants`]; that listing also extends [`VARIANT_NAMES`] and
/// [`datatype_representatives`], and this test then requires a fixture entry for it and rejects an
/// entry naming a variant that does not exist.
#[test]
fn the_frozen_text_covers_every_datatype_variant() {
let frozen = frozen_physical_types();
for data_type in datatype_representatives() {
let variant = variant_name(&data_type);
let entry = frozen
.iter()
.find(|entry| entry.variant == variant)
.unwrap_or_else(|| {
panic!("{variant}: v1_physical_types.json has no entry for this datatype variant")
});
// The entry's text is this variant's text: read it back and name the variant it describes.
let parsed: ConcreteDataType = serde_json::from_str(&entry.text)
.unwrap_or_else(|error| panic!("{variant}: {} is unreadable: {error}", entry.text));
assert_eq!(
variant_name(&parsed),
variant,
"{variant}: the entry's text describes this variant"
);
}
// Nothing in the fixture names a variant that does not exist, and no variant is listed twice.
for entry in &frozen {
assert!(
VARIANT_NAMES.contains(&entry.variant.as_str()),
"{}: the fixture names a variant ConcreteDataType does not have",
entry.variant
);
}
assert_eq!(
frozen.len(),
VARIANT_NAMES.len(),
"one fixture entry per datatype variant, no duplicates"
);
}