mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-15 07:17:01 +00:00
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
This commit is contained in:
@@ -15,10 +15,18 @@
|
||||
//! Streaming boundary between finite raw-series chunks and DataFusion record batches.
|
||||
//!
|
||||
//! Schema fields identify the ordered logical payload fields and codec layout; additional
|
||||
//! application metadata is permitted when stable across the stream. The caller-provided
|
||||
//! [`RawLabelSource`] is the label binding, not authenticated by the frame schema or field
|
||||
//! provenance. This adapter preserves scan candidates only; it is not a residual filter,
|
||||
//! authorization check, remote transport, or compaction boundary. Only
|
||||
//! application metadata is permitted when stable across the stream. A key-encoded frame also
|
||||
//! declares its own label decode description, and the frame is the sole source of decode
|
||||
//! information: [`unpack_raw_series`] reads the fields, the label mode, the description, and the
|
||||
//! producer ordering from the received schema, keeps the description on the returned stream, and
|
||||
//! therefore needs no [`RawLabelSource`]. A caller that holds a scan-bound identity — the physical
|
||||
//! region key layout a decoded identity came from — compares that identity's declared label
|
||||
//! decode layout with [`ensure_frame_matches_label_source`] instead of passing it as a decode
|
||||
//! input; that comparison is equality of the description alone and proves nothing about region
|
||||
//! identity or provenance, and neither side is authenticated as field provenance. This adapter
|
||||
//! preserves
|
||||
//! scan candidates only; it is not a residual filter, authorization check, remote transport, or
|
||||
//! compaction boundary. Only
|
||||
//! [`SeriesOrdering::Contiguous`] and [`SeriesOrdering::KeyThenTime`] are accepted;
|
||||
//! `Interleaved` is rejected before any poll.
|
||||
|
||||
@@ -32,14 +40,19 @@ use datafusion_common::DataFusionError;
|
||||
use datatypes::arrow::datatypes::{Schema as ArrowSchema, SchemaRef as ArrowSchemaRef};
|
||||
use datatypes::error::InvalidSeriesSnafu;
|
||||
use datatypes::series::frame::{
|
||||
LABEL_DECODE_VERSION, LabelDecodeColumn, LabelDecodeEncoding, LabelDecodeSpec, LabelDecodeTag,
|
||||
RawLabelMode, RawSeriesChunkCursor, RawSeriesFrameBuilder, RawSeriesFrameDecoder,
|
||||
raw_series_frame_schema,
|
||||
raw_series_frame_label_decode, raw_series_frame_schema_with_label_decode,
|
||||
raw_series_frame_series_schema,
|
||||
};
|
||||
use datatypes::series::raw::RawSeriesChunk;
|
||||
use datatypes::series::samples::SeriesSchema;
|
||||
use datatypes::series::{MetricName, SeriesOrdering};
|
||||
use futures_util::Stream;
|
||||
use store_api::region_engine::{RawLabelSource, RawSeriesStream, SendableRawSeriesStream};
|
||||
use store_api::codec::PrimaryKeyEncoding;
|
||||
use store_api::region_engine::{
|
||||
NO_BOUND_LABEL_SOURCE, RawLabelSource, RawSeriesStream, SendableRawSeriesStream,
|
||||
};
|
||||
|
||||
mod record_batches;
|
||||
|
||||
@@ -50,13 +63,35 @@ mod tests;
|
||||
|
||||
const ORDERING_KEY: &str = "greptime.series.ordering";
|
||||
|
||||
/// Builds the canonical raw frame schema with the producer's guaranteed ordering.
|
||||
/// Builds the canonical raw frame schema with the producer's label binding and ordering.
|
||||
///
|
||||
/// Ordering records a producer guarantee; it is not inferred from frame contents. `Interleaved`
|
||||
/// has no v1 wire representation and is rejected before the schema is assembled.
|
||||
/// has no v1 wire representation and is rejected before the schema is assembled. The label binding
|
||||
/// determines both the label mode and, for a primary-key binding, the self-describing label decode
|
||||
/// description written into the schema, so a frame can never declare a label layout it does not
|
||||
/// describe.
|
||||
pub fn raw_series_frame_schema_with_ordering(
|
||||
schema: &SeriesSchema,
|
||||
label_source: &RawLabelSource,
|
||||
ordering: SeriesOrdering,
|
||||
) -> datafusion_common::Result<ArrowSchemaRef> {
|
||||
frame_schema_with(
|
||||
schema,
|
||||
label_mode(label_source),
|
||||
label_decode_spec(label_source).as_ref(),
|
||||
ordering,
|
||||
)
|
||||
}
|
||||
|
||||
/// Assembles a declared frame schema from an explicit label mode and decode description.
|
||||
///
|
||||
/// The mode and description pair is checked by
|
||||
/// [`raw_series_frame_schema_with_label_decode`], so a key mode without a description and an
|
||||
/// explicit mode with one are both rejected here instead of reaching the wire.
|
||||
fn frame_schema_with(
|
||||
schema: &SeriesSchema,
|
||||
mode: RawLabelMode,
|
||||
decode: Option<&LabelDecodeSpec>,
|
||||
ordering: SeriesOrdering,
|
||||
) -> datafusion_common::Result<ArrowSchemaRef> {
|
||||
let ordering = match ordering {
|
||||
@@ -68,7 +103,8 @@ pub fn raw_series_frame_schema_with_ordering(
|
||||
));
|
||||
}
|
||||
};
|
||||
let schema = raw_series_frame_schema(schema, mode).map_err(codec_error)?;
|
||||
let schema =
|
||||
raw_series_frame_schema_with_label_decode(schema, mode, decode).map_err(codec_error)?;
|
||||
let mut metadata = schema.metadata().clone();
|
||||
metadata.insert(ORDERING_KEY.to_string(), ordering.to_string());
|
||||
Ok(Arc::new(ArrowSchema::new_with_metadata(
|
||||
@@ -77,6 +113,56 @@ pub fn raw_series_frame_schema_with_ordering(
|
||||
)))
|
||||
}
|
||||
|
||||
/// Derives the label decode description a frame must carry for this binding.
|
||||
///
|
||||
/// A primary-key binding reduces to exactly what decoding reads: the metric name, the primary-key
|
||||
/// encoding, the ordered key columns with their physical types, and the logical tags. Region
|
||||
/// metadata guarantees the key list holds every tag column and nothing else, so the description
|
||||
/// needs no per-column role. Explicit bindings already carry complete labels and declare no
|
||||
/// description.
|
||||
///
|
||||
/// The result is canonical: its logical tags are in [`LabelDecodeSpec::canonicalize`] order, which
|
||||
/// is what the writer serializes, so a derived description and a received one for the same labels
|
||||
/// are the same value and the same text. Derivation cannot fail, because a physical type is copied
|
||||
/// rather than translated; a description a reader cannot honour is rejected when it is read, and a
|
||||
/// decoder rejects the tags it cannot bind.
|
||||
pub fn label_decode_spec(source: &RawLabelSource) -> Option<LabelDecodeSpec> {
|
||||
let RawLabelSource::PrimaryKey {
|
||||
metric_name,
|
||||
physical_metadata,
|
||||
logical_tags,
|
||||
} = source
|
||||
else {
|
||||
return None;
|
||||
};
|
||||
let primary_key = physical_metadata
|
||||
.primary_key_columns()
|
||||
.map(|column| LabelDecodeColumn {
|
||||
column_id: column.column_id,
|
||||
name: column.column_schema.name.clone(),
|
||||
physical_type: column.column_schema.data_type.clone(),
|
||||
})
|
||||
.collect();
|
||||
let mut description = LabelDecodeSpec {
|
||||
version: LABEL_DECODE_VERSION,
|
||||
metric_name: metric_name.to_string(),
|
||||
encoding: match physical_metadata.primary_key_encoding {
|
||||
PrimaryKeyEncoding::Dense => LabelDecodeEncoding::Dense,
|
||||
PrimaryKeyEncoding::Sparse => LabelDecodeEncoding::Sparse,
|
||||
},
|
||||
primary_key,
|
||||
logical_tags: logical_tags
|
||||
.iter()
|
||||
.map(|tag| LabelDecodeTag {
|
||||
column_id: tag.column_id,
|
||||
name: tag.name.to_string(),
|
||||
})
|
||||
.collect(),
|
||||
};
|
||||
description.canonicalize();
|
||||
Some(description)
|
||||
}
|
||||
|
||||
/// Reads the declared producer ordering, conservatively treating legacy frames as contiguous.
|
||||
///
|
||||
/// Only the two orderings representable on the wire can be returned.
|
||||
@@ -99,17 +185,23 @@ pub fn raw_series_frame_ordering(
|
||||
/// work bound, not a resident-memory cap. A ready batch is yielded without polling the source
|
||||
/// again. Producers must keep each series' chunks contiguous. Validation checks adjacent chunk
|
||||
/// conditions but does not certify nonadjacent recurrence for `Contiguous` ordering.
|
||||
///
|
||||
/// The stream's own declaration is the only decode input: a frame unpacked without a scan-bound
|
||||
/// [`RawLabelSource`] round trips unchanged instead of being re-declared from a binding it does not
|
||||
/// have, and nothing is derived from a binding here. The declaration also decides the frame's label
|
||||
/// mode, so a chunk whose labels disagree with it is rejected when that chunk is coded.
|
||||
pub fn pack_raw_series(
|
||||
source: SendableRawSeriesStream,
|
||||
) -> datafusion_common::Result<DfSendableRecordBatchStream> {
|
||||
let ordering = source.ordering();
|
||||
ensure_supported_ordering(ordering)?;
|
||||
let schema = source.schema().clone();
|
||||
let label_source = source.label_source().clone();
|
||||
let mode = label_mode(&label_source);
|
||||
// The declared frame schema carries the producer ordering; the shared builder's schema
|
||||
// intentionally omits it, so every ready batch is declared before it is yielded.
|
||||
let frame_schema = raw_series_frame_schema_with_ordering(&schema, mode, ordering)?;
|
||||
let decode = source.label_decode_spec().cloned();
|
||||
let mode = declared_label_mode(&decode);
|
||||
// The declared frame schema carries the producer ordering and the label decode description;
|
||||
// the shared builder's schema intentionally omits both, so every ready batch is declared under
|
||||
// this schema before it is yielded, including when no row is ever produced.
|
||||
let frame_schema = frame_schema_with(&schema, mode, decode.as_ref(), ordering)?;
|
||||
let builder = RawSeriesFrameBuilder::try_new(
|
||||
&schema,
|
||||
mode,
|
||||
@@ -129,33 +221,48 @@ pub fn pack_raw_series(
|
||||
}))
|
||||
}
|
||||
|
||||
/// Unpacks DataFusion batches using the caller's explicit scan binding and ordering.
|
||||
/// Unpacks DataFusion batches declared by [`pack_raw_series`].
|
||||
///
|
||||
/// Only [`SeriesOrdering::Contiguous`] and [`SeriesOrdering::KeyThenTime`] streams are
|
||||
/// accepted; an `Interleaved` declaration is rejected before the stream is returned, including
|
||||
/// for an empty stream. Producers must keep each series' chunks contiguous. Validation checks
|
||||
/// adjacent chunk conditions but does not certify nonadjacent recurrence for `Contiguous`
|
||||
/// ordering.
|
||||
/// The frame is the sole source of decode information: the payload fields, the label mode, the
|
||||
/// label decode description, and the producer ordering are all read from the stream's own schema,
|
||||
/// so unpacking a frame a producer wrote needs no local copy of its binding. Only
|
||||
/// [`SeriesOrdering::Contiguous`] and [`SeriesOrdering::KeyThenTime`] streams are accepted; an
|
||||
/// `Interleaved` declaration is rejected before the stream is returned, including for an empty
|
||||
/// stream. Producers must keep each series' chunks contiguous. Validation checks adjacent chunk
|
||||
/// conditions but does not certify nonadjacent recurrence for `Contiguous` ordering.
|
||||
///
|
||||
/// The supplied binding is retained unchanged and is not validated as field provenance.
|
||||
/// Every frame is validated here, unconditionally. The declared layout must be this revision's, the
|
||||
/// label decode description must parse, name a known version, and agree with the declared label
|
||||
/// mode — a key frame must declare one and an explicit frame must not, since explicit labels are
|
||||
/// already complete. The declared fields and the empty batch are then decoded once, so a stream that
|
||||
/// could not be decoded is rejected before it is returned.
|
||||
///
|
||||
/// `expected` is a check, never an input. When it is given it must match the schema the frame
|
||||
/// declares — the same logical fields in the same order, and no evaluation grid — and a mismatch is
|
||||
/// an error. The decoder and the returned stream's schema are built from the frame either way, so a
|
||||
/// caller cannot supply a field layout the frame does not declare.
|
||||
pub fn unpack_raw_series(
|
||||
source: DfSendableRecordBatchStream,
|
||||
expected: Arc<SeriesSchema>,
|
||||
label_source: RawLabelSource,
|
||||
ordering: SeriesOrdering,
|
||||
expected: Option<Arc<SeriesSchema>>,
|
||||
) -> datafusion_common::Result<SendableRawSeriesStream> {
|
||||
ensure_supported_ordering(ordering)?;
|
||||
let mode = label_mode(&label_source);
|
||||
let frame_schema = source.schema();
|
||||
let decoder = RawSeriesFrameDecoder::try_new(&expected, mode).map_err(codec_error)?;
|
||||
let declared = raw_series_frame_label_decode(frame_schema.as_ref()).map_err(codec_error)?;
|
||||
let mode = declared_label_mode(&declared);
|
||||
let schema =
|
||||
Arc::new(raw_series_frame_series_schema(frame_schema.as_ref()).map_err(codec_error)?);
|
||||
if let Some(expected) = &expected {
|
||||
ensure_declared_schema(&schema, expected)?;
|
||||
}
|
||||
let ordering = raw_series_frame_ordering(frame_schema.as_ref())?;
|
||||
let decoder = RawSeriesFrameDecoder::try_new(&schema, mode).map_err(codec_error)?;
|
||||
// Validate even empty streams' declared schema and layout before returning the stream.
|
||||
let empty = DfRecordBatch::new_empty(frame_schema.clone());
|
||||
decoder.decode(&empty).map_err(codec_error)?;
|
||||
Ok(Box::pin(UnpackStream {
|
||||
source: Some(source),
|
||||
schema: expected,
|
||||
schema,
|
||||
frame_schema,
|
||||
label_source,
|
||||
label_decode_spec: declared,
|
||||
ordering,
|
||||
validator: SequenceValidator::new(ordering),
|
||||
pending: Vec::new().into_iter(),
|
||||
@@ -163,6 +270,66 @@ pub fn unpack_raw_series(
|
||||
}))
|
||||
}
|
||||
|
||||
/// Rejects an `expected` series schema the frame does not declare.
|
||||
///
|
||||
/// Only the two things a caller can get wrong are checked, because those are the only two things
|
||||
/// [`SeriesSchema`] holds: the logical fields and their order, and an evaluation grid a raw frame
|
||||
/// cannot carry.
|
||||
fn ensure_declared_schema(
|
||||
declared: &SeriesSchema,
|
||||
expected: &SeriesSchema,
|
||||
) -> datafusion_common::Result<()> {
|
||||
if expected.eval_grid.is_some() {
|
||||
return Err(invalid(
|
||||
"expected raw series schema declares an evaluation grid the frame cannot carry",
|
||||
));
|
||||
}
|
||||
if expected.fields.len() != declared.fields.len()
|
||||
|| expected
|
||||
.fields
|
||||
.iter()
|
||||
.zip(declared.fields.iter())
|
||||
.any(|(expected, declared)| expected.name != declared.name)
|
||||
{
|
||||
return Err(invalid(
|
||||
"raw frame fields do not match the expected series schema",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Checks that a frame declares the label decode layout a scan-bound binding describes.
|
||||
///
|
||||
/// This compares one thing: the description parsed from the frame's
|
||||
/// `greptime.series.label_decode` metadata against the description derived from `label_source`, by
|
||||
/// plain equality. A difference in the version, the metric name, the primary-key encoding, the
|
||||
/// ordered key columns and their physical types, or the logical tags is a mismatch. Nothing else is
|
||||
/// compared, so agreeing descriptions do **not** establish that these are the region's chunks: no
|
||||
/// region id, table id, or provenance is part of either side, and two regions whose tags have the
|
||||
/// same ids, names, and types produce equal descriptions. The check answers "does this frame describe
|
||||
/// the label layout this binding expects", never "is this the region I asked for".
|
||||
///
|
||||
/// It belongs to the caller that holds both sides: a frame describes its own labels, so
|
||||
/// [`unpack_raw_series`] decodes it without a binding, and a caller that received the frame from a
|
||||
/// peer compares the declaration here instead of decoding under whichever side it trusts.
|
||||
///
|
||||
/// The comparison is plain equality. It used to canonicalize logical tag order on both sides, which
|
||||
/// gave local comparisons one rule and the byte-exact remote comparison another; the writer now
|
||||
/// canonicalizes instead (see [`LabelDecodeSpec::canonicalize`]), so a description that reaches a
|
||||
/// reader already has one spelling and equality says everything there is to say about it.
|
||||
pub fn ensure_frame_matches_label_source(
|
||||
frame: &ArrowSchema,
|
||||
label_source: &RawLabelSource,
|
||||
) -> datafusion_common::Result<()> {
|
||||
let declared = raw_series_frame_label_decode(frame).map_err(codec_error)?;
|
||||
if declared != label_decode_spec(label_source) {
|
||||
return Err(invalid(
|
||||
"raw frame label decode description does not match the supplied binding",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn label_mode(source: &RawLabelSource) -> RawLabelMode {
|
||||
match source {
|
||||
RawLabelSource::PrimaryKey { .. } => RawLabelMode::Key,
|
||||
@@ -170,6 +337,15 @@ fn label_mode(source: &RawLabelSource) -> RawLabelMode {
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the label mode a frame declares: a key frame describes its key, an explicit frame
|
||||
/// carries complete labels and declares no description.
|
||||
fn declared_label_mode(declared: &Option<LabelDecodeSpec>) -> RawLabelMode {
|
||||
match declared {
|
||||
Some(_) => RawLabelMode::Key,
|
||||
None => RawLabelMode::Explicit,
|
||||
}
|
||||
}
|
||||
|
||||
/// Rejects orderings this adapter cannot validate; a bad declaration is an error, not a fallback.
|
||||
fn ensure_supported_ordering(ordering: SeriesOrdering) -> datafusion_common::Result<()> {
|
||||
match ordering {
|
||||
@@ -408,7 +584,8 @@ struct UnpackStream {
|
||||
source: Option<DfSendableRecordBatchStream>,
|
||||
schema: Arc<SeriesSchema>,
|
||||
frame_schema: datatypes::arrow::datatypes::SchemaRef,
|
||||
label_source: RawLabelSource,
|
||||
/// The description parsed from the frame schema; the only input decoding needs.
|
||||
label_decode_spec: Option<LabelDecodeSpec>,
|
||||
ordering: SeriesOrdering,
|
||||
validator: SequenceValidator,
|
||||
pending: std::vec::IntoIter<RawSeriesChunk>,
|
||||
@@ -420,8 +597,18 @@ impl RawSeriesStream for UnpackStream {
|
||||
&self.schema
|
||||
}
|
||||
|
||||
/// Returns [`NO_BOUND_LABEL_SOURCE`]: a frame establishes its own label layout and carries no
|
||||
/// scan-bound identity, so decoding reads [`Self::label_decode_spec`] and never this. There is
|
||||
/// no physical region key layout to report and none is invented from the description, which
|
||||
/// deliberately carries no region metadata.
|
||||
fn label_source(&self) -> &RawLabelSource {
|
||||
&self.label_source
|
||||
&NO_BOUND_LABEL_SOURCE
|
||||
}
|
||||
|
||||
/// Returns the label decode description the frame declared, which is what decoding reads; an
|
||||
/// explicit-label frame declares none.
|
||||
fn label_decode_spec(&self) -> Option<&LabelDecodeSpec> {
|
||||
self.label_decode_spec.as_ref()
|
||||
}
|
||||
|
||||
fn ordering(&self) -> SeriesOrdering {
|
||||
|
||||
@@ -1130,13 +1130,7 @@ async fn exact_int64_filter_sort_adapter_frame_pipeline() {
|
||||
.into(),
|
||||
eval_grid: None,
|
||||
});
|
||||
let mut decoded = super::super::unpack_raw_series(
|
||||
frames,
|
||||
expected,
|
||||
store_api::region_engine::RawLabelSource::Explicit,
|
||||
SeriesOrdering::Contiguous,
|
||||
)
|
||||
.unwrap();
|
||||
let mut decoded = super::super::unpack_raw_series(frames, Some(expected)).unwrap();
|
||||
let chunk = decoded.next().await.unwrap().unwrap();
|
||||
assert!(chunk.is_last);
|
||||
assert_eq!(chunk.time.as_buffer().as_ref(), &[1, 2]);
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -193,13 +193,7 @@ async fn streaming_round_trip_preserves_sliced_multifield_payload_and_owners() {
|
||||
chunk: Some(chunk),
|
||||
});
|
||||
let frames = pack_raw_series(input).unwrap();
|
||||
let mut decoded = unpack_raw_series(
|
||||
frames,
|
||||
schema,
|
||||
RawLabelSource::Explicit,
|
||||
SeriesOrdering::Contiguous,
|
||||
)
|
||||
.unwrap();
|
||||
let mut decoded = unpack_raw_series(frames, Some(schema)).unwrap();
|
||||
let mut output = Vec::new();
|
||||
while let Some(item) = decoded.next().await {
|
||||
output.push(item.unwrap());
|
||||
@@ -425,13 +419,7 @@ async fn single_row_sliced_payload_keeps_zero_copy_buffers_and_releases_owners()
|
||||
chunk: Some(chunk),
|
||||
});
|
||||
let frames = pack_raw_series(input).unwrap();
|
||||
let mut decoded = unpack_raw_series(
|
||||
frames,
|
||||
schema,
|
||||
RawLabelSource::Explicit,
|
||||
SeriesOrdering::Contiguous,
|
||||
)
|
||||
.unwrap();
|
||||
let mut decoded = unpack_raw_series(frames, Some(schema)).unwrap();
|
||||
let mut output = Vec::new();
|
||||
while let Some(item) = decoded.next().await {
|
||||
output.push(item.unwrap());
|
||||
|
||||
Reference in New Issue
Block a user