mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 23:06:59 +00:00
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
This commit is contained in:
discord9
@@ -31,6 +31,7 @@ use datafusion_physical_plan::metrics::ExecutionPlanMetricsSet;
|
||||
use datafusion_physical_plan::{DisplayAs, DisplayFormatType, PhysicalExpr};
|
||||
use datatypes::schema::SchemaRef;
|
||||
use datatypes::series::SeriesOrdering;
|
||||
use datatypes::series::frame::LabelDecodeSpec;
|
||||
use datatypes::series::raw::RawSeriesChunk;
|
||||
use datatypes::series::samples::SeriesSchema;
|
||||
use futures::Stream;
|
||||
@@ -495,10 +496,20 @@ pub enum RawLabelSource {
|
||||
},
|
||||
/// The stream supplies a complete metric name and label set in each chunk, not a partial
|
||||
/// label projection; `Some(metric)` is a complete name/label set (the name itself may be
|
||||
/// absent).
|
||||
/// absent), or, for a stream that has no identity at all, see
|
||||
/// [`NO_BOUND_LABEL_SOURCE`].
|
||||
Explicit,
|
||||
}
|
||||
|
||||
/// The identity a stream reports when no scan-bound label source was established for it.
|
||||
///
|
||||
/// [`RawLabelSource`] has no "absent" variant, and [`RawLabelSource::Explicit`] is the only one that
|
||||
/// carries no physical region key layout, so it stands in for "no identity" here. It is not a claim
|
||||
/// that the chunks carry labels: [`RawSeriesStream::label_decode_spec`] answers that, and a stream
|
||||
/// reporting this value can still return `Some` description and leave `metric` as `None` in every
|
||||
/// chunk. Decoding therefore asks the description, and only an identity check asks this.
|
||||
pub const NO_BOUND_LABEL_SOURCE: RawLabelSource = RawLabelSource::Explicit;
|
||||
|
||||
/// A stream of raw series chunks with stable schema, label binding, and ordering.
|
||||
///
|
||||
/// Every series has strictly increasing timestamps within and across chunks; timestamps are
|
||||
@@ -508,8 +519,10 @@ pub enum RawLabelSource {
|
||||
/// per-stream guarantees, not query-global order, partition ownership, or finality. `is_last`
|
||||
/// marks the last chunk of that series in this stream only; merging requires recomputing
|
||||
/// per-input-series finalization across all related inputs, not aggregate-group finality.
|
||||
/// `RawLabelSource` keeps label interpretation self-contained through transport. Stream finality
|
||||
/// applies to all related input series, not aggregate groups or vector-match state.
|
||||
/// Label interpretation is carried by the stream: [`RawSeriesStream::label_source`] states the
|
||||
/// scan-bound identity a producer read under, and [`RawSeriesStream::label_decode_spec`] states the
|
||||
/// label layout the stream's own chunks declare. Stream finality applies to all related input
|
||||
/// series, not aggregate groups or vector-match state.
|
||||
pub trait RawSeriesStream: Stream<Item = Result<RawSeriesChunk, BoxedError>> + Send {
|
||||
/// Returns the schema shared by all chunks; raw schemas have no evaluation grid.
|
||||
fn schema(&self) -> &Arc<SeriesSchema>;
|
||||
@@ -517,7 +530,26 @@ pub trait RawSeriesStream: Stream<Item = Result<RawSeriesChunk, BoxedError>> + S
|
||||
/// labels are undecoded, not that the metric name is missing; `Explicit` chunks supply
|
||||
/// `Some` with the complete name/label set, not a partial projection. The metric name itself
|
||||
/// may be absent.
|
||||
///
|
||||
/// A stream that was decoded from a raw frame has no scan-bound identity and reports
|
||||
/// [`NO_BOUND_LABEL_SOURCE`], which reads as `Explicit` while its chunks may still carry no
|
||||
/// labels at all. That value therefore answers "is this the region I read under", never "are
|
||||
/// these labels complete": [`Self::label_decode_spec`] is the stream's statement about its
|
||||
/// encoding, and every consumer that decides whether to decode reads that instead.
|
||||
fn label_source(&self) -> &RawLabelSource;
|
||||
/// Returns the label decode description this stream declares for key-encoded labels.
|
||||
///
|
||||
/// Every stream whose chunks carry key-encoded labels declares the description here, whether
|
||||
/// it is a storage producer or a stream reassembled from a raw frame. A consumer holding only
|
||||
/// that description can therefore decode labels without a scan-bound [`RawLabelSource`]: the
|
||||
/// description, not the binding, describes the encoding, and a receiver reporting
|
||||
/// [`NO_BOUND_LABEL_SOURCE`] still states through this description that its chunks are
|
||||
/// undecoded key chunks. A stream that carries complete labels instead answers `None`. This is
|
||||
/// the description the stream declares, not an authenticated statement about where the values
|
||||
/// came from.
|
||||
fn label_decode_spec(&self) -> Option<&LabelDecodeSpec> {
|
||||
None
|
||||
}
|
||||
/// Returns this stream's ordering guarantee.
|
||||
fn ordering(&self) -> SeriesOrdering;
|
||||
}
|
||||
|
||||
@@ -26,11 +26,15 @@ use datatypes::error::InvalidSeriesSnafu;
|
||||
use datatypes::prelude::ConcreteDataType;
|
||||
use datatypes::schema::ColumnSchema;
|
||||
use datatypes::series::SeriesOrdering;
|
||||
use datatypes::series::frame::{
|
||||
LABEL_DECODE_VERSION, LabelDecodeColumn, LabelDecodeEncoding, LabelDecodeSpec, LabelDecodeTag,
|
||||
};
|
||||
use datatypes::series::raw::{RawSeriesChunk, RawTimestamps};
|
||||
use datatypes::series::samples::{FieldSpec, Samples, SeriesFields, SeriesSchema};
|
||||
use datatypes::series::{Label, MetricName};
|
||||
use futures::{Stream, StreamExt};
|
||||
|
||||
use crate::codec::PrimaryKeyEncoding;
|
||||
use crate::region_engine::{
|
||||
PreparedSeriesSource, RawLabelSource, RawSeriesStream, SendableRawSeriesStream, SeriesInfo,
|
||||
};
|
||||
@@ -91,6 +95,7 @@ impl PreparedSeriesSource for MemorySource {
|
||||
Ok(Box::pin(MemoryReader {
|
||||
chunks,
|
||||
schema: self.schema.clone(),
|
||||
label_decode_spec: declared_decode_spec(&self.label_source),
|
||||
label_source: self.label_source.clone(),
|
||||
polls: self.polls.clone(),
|
||||
drops: self.drops.clone(),
|
||||
@@ -101,6 +106,8 @@ impl PreparedSeriesSource for MemorySource {
|
||||
struct MemoryReader {
|
||||
chunks: VecDeque<RawSeriesChunk>,
|
||||
schema: Arc<SeriesSchema>,
|
||||
/// The description this fixture declares, derived once from `label_source`, as a producer does.
|
||||
label_decode_spec: Option<LabelDecodeSpec>,
|
||||
label_source: RawLabelSource,
|
||||
polls: Arc<AtomicUsize>,
|
||||
drops: Arc<AtomicUsize>,
|
||||
@@ -133,11 +140,58 @@ impl RawSeriesStream for MemoryReader {
|
||||
&self.label_source
|
||||
}
|
||||
|
||||
fn label_decode_spec(&self) -> Option<&LabelDecodeSpec> {
|
||||
self.label_decode_spec.as_ref()
|
||||
}
|
||||
|
||||
fn ordering(&self) -> SeriesOrdering {
|
||||
SeriesOrdering::Contiguous
|
||||
}
|
||||
}
|
||||
|
||||
/// Describes the key-encoded chunks a reader of `source` yields.
|
||||
///
|
||||
/// This crate cannot depend on `common-query` (which depends on it), so this fixture repeats the
|
||||
/// derivation `common_query::series_frame::label_decode_spec` performs for a producer: the metric
|
||||
/// name, the primary-key encoding, the ordered key columns with their physical types, and the
|
||||
/// logical tags in canonical order. An explicit binding already carries complete labels and
|
||||
/// declares nothing.
|
||||
fn declared_decode_spec(source: &RawLabelSource) -> Option<LabelDecodeSpec> {
|
||||
let RawLabelSource::PrimaryKey {
|
||||
metric_name,
|
||||
physical_metadata,
|
||||
logical_tags,
|
||||
} = source
|
||||
else {
|
||||
return None;
|
||||
};
|
||||
let mut description = LabelDecodeSpec {
|
||||
version: LABEL_DECODE_VERSION,
|
||||
metric_name: metric_name.to_string(),
|
||||
encoding: match physical_metadata.primary_key_encoding {
|
||||
PrimaryKeyEncoding::Dense => LabelDecodeEncoding::Dense,
|
||||
PrimaryKeyEncoding::Sparse => LabelDecodeEncoding::Sparse,
|
||||
},
|
||||
primary_key: physical_metadata
|
||||
.primary_key_columns()
|
||||
.map(|column| LabelDecodeColumn {
|
||||
column_id: column.column_id,
|
||||
name: column.column_schema.name.clone(),
|
||||
physical_type: column.column_schema.data_type.clone(),
|
||||
})
|
||||
.collect(),
|
||||
logical_tags: logical_tags
|
||||
.iter()
|
||||
.map(|tag| LabelDecodeTag {
|
||||
column_id: tag.column_id,
|
||||
name: tag.name.to_string(),
|
||||
})
|
||||
.collect(),
|
||||
};
|
||||
description.canonicalize();
|
||||
Some(description)
|
||||
}
|
||||
|
||||
fn metric(host: &str) -> MetricName {
|
||||
MetricName::try_new(
|
||||
Some(Arc::from("cpu")),
|
||||
@@ -358,6 +412,25 @@ async fn prepared_source_label_binding_is_available_before_readers_and_matches_c
|
||||
|
||||
let reader = source.read_series(0..2).unwrap();
|
||||
assert_same_label_source(source.label_source(), reader.label_source());
|
||||
match (explicit, reader.label_decode_spec()) {
|
||||
(true, None) => {}
|
||||
(false, Some(description)) => {
|
||||
assert_eq!(description.version, LABEL_DECODE_VERSION);
|
||||
assert_eq!(description.metric_name, "cpu");
|
||||
assert_eq!(description.encoding, LabelDecodeEncoding::Dense);
|
||||
assert_eq!(description.primary_key.len(), 1);
|
||||
assert_eq!(description.primary_key[0].column_id, 1);
|
||||
assert_eq!(description.primary_key[0].name, "host");
|
||||
assert_eq!(
|
||||
description.primary_key[0].physical_type,
|
||||
ConcreteDataType::string_datatype()
|
||||
);
|
||||
assert_eq!(description.logical_tags.len(), 1);
|
||||
assert_eq!(description.logical_tags[0].column_id, 1);
|
||||
assert_eq!(description.logical_tags[0].name, "host");
|
||||
}
|
||||
_ => panic!("unexpected declared label decode description"),
|
||||
}
|
||||
assert_eq!(source.polls.load(Ordering::SeqCst), 0);
|
||||
let chunks = futures::TryStreamExt::try_collect::<Vec<_>>(reader)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user