rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-09-05 13:08:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: support raw OTLP delta metrics (#8970)

Browse Source 
* feat: support raw OTLP delta metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: fmt

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test(promql): update sqlness results for normalized label matching

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: derive temporality label from default column prefix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test(promql): add analyze coverage for delta temporality

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): scope label alignment to temporality marker

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: handle count-only histograms and vector broadcasts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude temporality marker from entity descriptions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: use a fixed label for OTLP aggregation temporality

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): preserve mixed-range semantics for raw delta

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
This commit is contained in:
shuiyisong
2026-09-03 09:28:11 +00:00
committed by GitHub
parent 945e53e0a3
commit b86da3d35f
24 changed files with 3266 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfcs/2026-05-28-table-semantic-layer.md
+53 -1
View File
@@ -78,7 +78,7 @@ concern, not query-time semantics).
| --- | --- |
| `greptime.semantic.metric.type` | `counter` / `gauge` / `histogram` / `summary` / `updown_counter` / `gauge_histogram` / `info` / `stateset` |
| `greptime.semantic.metric.unit` | UCUM, e.g. `s`, `By`, `{request}` (discarded by the row encoders, so unrecoverable once ingested) |
| `greptime.semantic.metric.temporality` | `cumulative` / `delta` (OTel only; invisible in the name) |
| `greptime.semantic.metric.temporality` | `cumulative` / `delta` / `mixed` (OTel only; catalog-level description) |
| `greptime.semantic.metric.metadata_quality` | `declared` (OTLP / exposition) or `inferred` (Prom RW v1, name-suffix guess) |
| `greptime.semantic.metric.original_name` | Pre-translation OTel name when the table name was Prometheus-ised; the key a consumer uses to look the metric up in the OTel semantic conventions |
@@ -93,6 +93,58 @@ Two design decisions worth pinning down up front, because they constrain everyth
- **Conflict.** Some table-level keys (`trace.conventions` lifted from `schema_url`, `metric.temporality`, ...) cannot represent the truth when a long-lived table sees rows from multiple sources. v1 records `mixed` or `unknown` rather than a fictitious single value. Downstream consumers must treat any single-valued semantic key as best-effort, not strong evidence.
- **Update.** Semantic options are stamped at table creation. v1 does not specify an update path; promoting `metadata_quality` from `inferred` to `declared`, refreshing `resource.attributes_preserved`, or revising `trace.conventions` on later writes is deferred. If real usage shows update is needed, it lands as a separate RFC.
OTLP delta sums and explicit histograms additionally store the query-visible
String tag `otlp_aggregation_temporality="delta"` on each generated row. Its
name is fixed and does not follow `default_column_prefix`. The tag is part of
series identity and is authoritative for per-series float `rate()` and
`increase()` behavior; the table option is never used as a row-level
discriminator. Native histograms retain their native algorithms. Prometheus
metadata reports `unknown` for counter, histogram, and up/down-counter tables
whose catalog temporality is `delta` or `mixed`. A same-request conflict can
create `mixed`; a later write does not update an existing table option, so the
catalog value can also be stale while the row tag remains authoritative.
The concrete mixed-temporality workload is a rolling production change from
cumulative to delta for the same metric name. Old and new exporters can overlap
during rollout, retries and late points can extend that overlap, and retained
cumulative history must remain queryable after the fleet converges. Rejecting a
different temporality at table level therefore prevents an in-place transition.
Routing delta rows to another table either exposes a different metric/table to
users or requires a logical union that still needs a per-series temporality
discriminator. v1 keeps one table and stores that discriminator on the series.
The marker remains in PromQL results and follows ordinary label matching and
grouping. The supported states of this reserved label are absent/NULL
(cumulative) and `delta`. Exact Metric Engine arithmetic and comparison matching
reuses the existing `__tsid` key, so mixed-temporality support adds no projected
matching columns on that path. When `__tsid` is unavailable and the marker
participates in matching, the planner aligns only
`otlp_aggregation_temporality`, projecting a nullable String on an input whose
schema lacks it; unrelated nullable labels retain their existing matching
behavior. Ignoring the marker adds no marker-related matching work.
Use `ignoring(otlp_aggregation_temporality)` when temporality should not
participate in matching. Apply `rate()` or `increase()` before an aggregation
or subquery that drops the marker. `irate()` and `resets()` are not
temporality-aware in v1; `delta()`, `idelta()`, and `changes()` retain their
existing raw-sample semantics.
`otlp_aggregation_temporality` is a reserved stored key for OTLP metric
attributes. Existing float tables that already contain that exact String tag
and the value `delta` opt into this behavior after upgrade. Producers should
rename such a user-defined label if that was not their intent. A non-OTLP writer
may opt in deliberately, while a namespaced OTLP scope attribute cannot collide
with the stored key.
OTLP `NoRecordedValue` sums store the canonical Prometheus stale marker.
Classic-histogram tombstones mark the point's supplied bounds, implicit `+Inf`,
`_count`, and `_sum` when supplied. If the optional `sum` is absent, no `_sum`
stale marker is emitted, so a previously stored `_sum` sample can remain
visible to instant selectors until lookback expiry while `_count` and the
supplied bucket series are stale. Bounds absent from or changed on the
tombstone can likewise remain visible; complete marking would require retained
per-stream component and bound-layout history and is outside v1.
## `information_schema.table_semantics`
A consumer's first SQL on connect:
src/common/query/src/prelude.rs
+4
View File
@@ -93,6 +93,10 @@ const GREPTIME_TIMESTAMP: &str = "greptime_timestamp";
const GREPTIME_VALUE: &str = "greptime_value";
/// Default counter column name for OTLP metrics (legacy mode).
pub const GREPTIME_COUNT: &str = "greptime_count";
/// Stored series label that opts ordinary float samples into raw-delta math.
pub const OTLP_AGGREGATION_TEMPORALITY_LABEL: &str = "otlp_aggregation_temporality";
/// Authoritative raw-delta value for [`OTLP_AGGREGATION_TEMPORALITY_LABEL`].
pub const GREPTIME_TEMPORALITY_DELTA: &str = "delta";
/// Default physical table name
pub const GREPTIME_PHYSICAL_TABLE: &str = "greptime_physical_table";
src/frontend/src/instance/entity_graph.rs
+10 -1
View File
@@ -42,6 +42,7 @@ use common_catalog::consts::{
use common_error::ext::{BoxedError, ErrorExt};
use common_error::status_code::StatusCode;
use common_query::OutputData;
use common_query::prelude::OTLP_AGGREGATION_TEMPORALITY_LABEL;
use common_recordbatch::SendableRecordBatchStream;
use common_telemetry::{debug, warn};
use common_time::timestamp::TimeUnit;
@@ -342,6 +343,7 @@ impl EntityGraphProviderImpl {
.meta
.row_key_column_names()
.filter(|c| !implicit.id.contains(c))
.filter(|c| c.as_str() != OTLP_AGGREGATION_TEMPORALITY_LABEL)
.cloned()
.collect()
} else {
@@ -1249,9 +1251,16 @@ mod tests {
#[test]
fn target_info_descriptive_rest_covers_remaining_tags() {
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
let info = prom_table_info(
"target_info",
&["job", "instance", "k8s_cluster_name", "service_version"],
&[
"job",
"instance",
"k8s_cluster_name",
"service_version",
marker,
],
PROM_STAMPS,
);
let declarations = sorted_declarations(&info);
src/frontend/src/instance/otlp.rs
+1 -1
View File
@@ -130,7 +130,7 @@ impl OpenTelemetryProtocolHandler for Instance {
} = otlp::metrics::to_grpc_insert_requests(request, &mut metric_ctx)?;
if outcome.rejected_data_points > 0 {
warn!(
"Rejected {} OTLP exponential histogram data points: {}",
"Rejected {} OTLP metrics data points: {}",
outcome.rejected_data_points,
outcome.error_message.as_deref().unwrap_or_default()
);
src/promql/benches/bench_range_fn.rs
+101 -1
View File
@@ -37,7 +37,7 @@ use datatypes::arrow::datatypes::{DataType, Field};
use futures::StreamExt;
use promql::extension_plan::RangeManipulate;
use promql::functions::{
Changes, Delta, IDelta, Increase, PredictLinear, QuantileOverTime, Rate, Resets,
Changes, Delta, IDelta, Increase, PredictLinear, QuantileOverTime, Rate, Resets, SumOverTime,
};
use promql::range_array::RangeArray;
@@ -135,6 +135,62 @@ fn make_extrapolated_rate_input(
]
}
fn make_delta_rate_comparison_input(
series_count: usize,
hours: usize,
sample_step_seconds: usize,
query_step_seconds: usize,
window_seconds: usize,
) -> (Vec<ColumnarValue>, Vec<ColumnarValue>) {
let points_per_series = hours * 60 * 60 / sample_step_seconds;
let window_points = window_seconds / sample_step_seconds;
let query_stride = query_step_seconds / sample_step_seconds;
let mut timestamps = Vec::with_capacity(series_count * points_per_series);
let mut deltas = Vec::with_capacity(timestamps.capacity());
let mut cumulative = Vec::with_capacity(timestamps.capacity());
let mut ranges = Vec::new();
let mut eval_timestamps = Vec::new();
for _ in 0..series_count {
let offset = timestamps.len();
let mut total = 0.0;
for point in 0..points_per_series {
let delta = 1.0 + (point % 7) as f64 * 0.25;
total += delta;
timestamps.push((point as i64 + 1) * sample_step_seconds as i64 * 1_000);
deltas.push(delta);
cumulative.push(total);
}
for end in (window_points - 1..points_per_series).step_by(query_stride) {
ranges.push((
(offset + end + 1 - window_points) as u32,
window_points as u32,
));
eval_timestamps.push(timestamps[offset + end] + 500);
}
}
let timestamps = Arc::new(TimestampMillisecondArray::from(timestamps));
let delta_timestamp_ranges =
RangeArray::from_ranges(timestamps.clone(), ranges.clone()).unwrap();
let cumulative_timestamp_ranges = RangeArray::from_ranges(timestamps, ranges.clone()).unwrap();
let delta_ranges =
RangeArray::from_ranges(Arc::new(Float64Array::from(deltas)), ranges.clone()).unwrap();
let cumulative_ranges =
RangeArray::from_ranges(Arc::new(Float64Array::from(cumulative)), ranges).unwrap();
let delta = vec![
ColumnarValue::Array(Arc::new(delta_timestamp_ranges.into_dict())),
ColumnarValue::Array(Arc::new(delta_ranges.into_dict())),
];
let cumulative = vec![
ColumnarValue::Array(Arc::new(cumulative_timestamp_ranges.into_dict())),
ColumnarValue::Array(Arc::new(cumulative_ranges.into_dict())),
ColumnarValue::Array(Arc::new(TimestampMillisecondArray::from(eval_timestamps))),
ColumnarValue::Scalar(ScalarValue::Int64(Some(window_seconds as i64 * 1_000))),
];
(delta, cumulative)
}
fn make_idelta_input(num_points: usize, window_size: u32) -> Vec<ColumnarValue> {
let (ts_range, val_range, _) =
build_sliding_ranges(num_points, window_size, build_default_values(num_points), 0);
@@ -453,6 +509,49 @@ fn bench_range_functions(c: &mut Criterion) {
group.finish();
}
fn bench_delta_rate_comparison(c: &mut Criterion) {
let mut group = c.benchmark_group("delta_rate_comparison");
let series_count = 64;
let hours = 4;
let sample_step_seconds = 15;
let window_seconds = 2 * 60 * 60;
let delta_udf = SumOverTime::scalar_udf();
let cumulative_udf = Rate::scalar_udf();
// Release acceptance threshold: on both step sweeps, the sum reducer that
// dominates delta-rate cost should stay below 100 ms and within 100x of
// cumulative rate on this 64-series, four-hour data set. Optimize the
// reducer before release if either bound is exceeded on a typical CI host.
for query_step_seconds in [60, 300] {
let (delta, cumulative) = make_delta_rate_comparison_input(
series_count,
hours,
sample_step_seconds,
query_step_seconds,
window_seconds,
);
let delta = PreparedUdfCall::new(delta);
let cumulative = PreparedUdfCall::new(cumulative);
let parameters = format!(
"series{series_count}_hours{hours}_window{}h_step{}s",
window_seconds / 60 / 60,
query_step_seconds
);
group.bench_with_input(
BenchmarkId::new("delta_sum_over_time", &parameters),
&(),
|b, _| b.iter(|| invoke_prepared(&delta_udf, &delta)),
);
group.bench_with_input(
BenchmarkId::new("cumulative_rate", &parameters),
&(),
|b, _| b.iter(|| invoke_prepared(&cumulative_udf, &cumulative)),
);
}
group.finish();
}
fn bench_edge_count_functions(c: &mut Criterion) {
let mut group = c.benchmark_group("edge_count_fn");
let num_points = 4_096;
@@ -781,6 +880,7 @@ fn bench_range_manipulate_wall_time(c: &mut Criterion) {
criterion_group!(
benches,
bench_range_functions,
bench_delta_rate_comparison,
bench_edge_count_functions,
bench_range_manipulate_wall_time
);
src/promql/src/functions/native_histogram.rs
+16 -3
View File
@@ -1548,6 +1548,9 @@ impl ScalarUDFImpl for MixedRangeUdf {
enum MixedRangeFunction {
Rate,
Increase,
// Raw-delta modes sum floats while preserving mixed-range drop/warning semantics.
RawDeltaRate,
RawDeltaIncrease,
Delta,
IDelta,
IRate,
@@ -1587,6 +1590,8 @@ impl MixedRangeFunction {
match name {
"rate" => Ok(Self::Rate),
"increase" => Ok(Self::Increase),
"raw_delta_rate" => Ok(Self::RawDeltaRate),
"raw_delta_increase" => Ok(Self::RawDeltaIncrease),
"delta" => Ok(Self::Delta),
"idelta" => Ok(Self::IDelta),
"irate" => Ok(Self::IRate),
@@ -1617,6 +1622,8 @@ impl MixedRangeFunction {
match self {
Self::Rate => "rate",
Self::Increase => "increase",
Self::RawDeltaRate => "rate",
Self::RawDeltaIncrease => "increase",
Self::Delta => "delta",
Self::IDelta => "idelta",
Self::IRate => "irate",
@@ -1642,9 +1649,13 @@ impl MixedRangeFunction {
fn policy(self) -> MixedRangePolicy {
match self {
Self::Rate | Self::Increase | Self::Delta | Self::AvgOverTime | Self::SumOverTime => {
MixedRangePolicy::DropMixed
}
Self::Rate
| Self::Increase
| Self::RawDeltaRate
| Self::RawDeltaIncrease
| Self::Delta
| Self::AvgOverTime
| Self::SumOverTime => MixedRangePolicy::DropMixed,
Self::IDelta | Self::IRate => MixedRangePolicy::LastTwo,
Self::LastOverTime => MixedRangePolicy::Last,
Self::Changes
@@ -1668,6 +1679,7 @@ impl MixedRangeFunction {
match self {
Self::Rate => Some(Rate::scalar_udf()),
Self::Increase => Some(Increase::scalar_udf()),
Self::RawDeltaRate | Self::RawDeltaIncrease => Some(SumOverTime::scalar_udf()),
Self::Delta => Some(crate::functions::Delta::scalar_udf()),
Self::IDelta => Some(IDelta::<false>::scalar_udf()),
Self::IRate => Some(IDelta::<true>::scalar_udf()),
@@ -1708,6 +1720,7 @@ impl MixedRangeFunction {
collector,
)),
Self::LastOverTime => Some(NativeHistogramLastOverTime::scalar_udf()),
Self::RawDeltaRate | Self::RawDeltaIncrease => None,
_ => None,
}
}
src/query/src/promql/planner.rs
+471 -103
View File
@@ -23,7 +23,10 @@ use common_error::ext::ErrorExt;
use common_error::status_code::StatusCode;
use common_function::function::FunctionContext;
use common_query::native_histogram::native_histogram_value_type;
use common_query::prelude::{greptime_native_histogram, greptime_value};
use common_query::prelude::{
GREPTIME_TEMPORALITY_DELTA, OTLP_AGGREGATION_TEMPORALITY_LABEL, greptime_native_histogram,
greptime_value,
};
use common_query::promql_annotations::PromqlAnnotationCollector;
use datafusion::common::DFSchemaRef;
use datafusion::datasource::DefaultTableSource;
@@ -1062,8 +1065,14 @@ impl PromPlanner {
first_leaf: &PlannedIslandLeaf,
right_leaf: &PlannedIslandLeaf,
) -> Result<LogicalPlan> {
let only_join_time_index =
first_leaf.ctx.tag_columns.is_empty() || right_leaf.ctx.tag_columns.is_empty();
let only_join_time_index = (first_leaf.ctx.tag_columns.is_empty()
|| right_leaf.ctx.tag_columns.is_empty())
&& !first_leaf
.ctx
.tag_columns
.iter()
.chain(&right_leaf.ctx.tag_columns)
.any(|tag| tag == OTLP_AGGREGATION_TEMPORALITY_LABEL);
let (mut left_keys, mut right_keys, force_empty_join) = self.binary_join_key_columns(
left.schema(),
right_leaf.plan.schema(),
@@ -1502,6 +1511,8 @@ impl PromPlanner {
.table_ref()
.unwrap_or_else(|_| TableReference::bare(""));
let right_context = self.ctx.clone();
let left_is_empty_metric = Self::is_empty_metric(&left_input);
let right_is_empty_metric = Self::is_empty_metric(&right_input);
// TODO(ruihang): avoid join if left and right are the same table
@@ -1541,6 +1552,8 @@ impl PromPlanner {
} else {
self.ctx.table_name = Some("rhs".to_string());
}
} else if right_is_empty_metric && !left_is_empty_metric {
self.ctx = left_context.clone();
}
// Computed scalars reach this join path instead of the literal projection paths.
// Broadcast them for arithmetic in the same way as literal scalars.
@@ -1577,6 +1590,8 @@ impl PromPlanner {
.map(|(output, _)| output.clone())
.collect();
let mut field_groups = field_groups.into_iter();
// `vector()` uses EmptyMetric and keeps GreptimeDB's timestamp broadcast.
let has_empty_metric_operand = left_is_empty_metric || right_is_empty_metric;
let join_plan = self.join_on_non_field_columns(
left_input,
@@ -1585,9 +1600,16 @@ impl PromPlanner {
right_table_ref.clone(),
left_time_index_column,
right_time_index_column,
// if left plan or right plan tag is empty, means case like `scalar(...) + host` or `host + scalar(...)`
// under this case we only join on time index
left_context.tag_columns.is_empty() || right_context.tag_columns.is_empty(),
lhs.value_type() == ValueType::Scalar
|| rhs.value_type() == ValueType::Scalar
|| has_empty_metric_operand
|| ((left_context.tag_columns.is_empty()
|| right_context.tag_columns.is_empty())
&& !left_context
.tag_columns
.iter()
.chain(&right_context.tag_columns)
.any(|tag| tag == OTLP_AGGREGATION_TEMPORALITY_LABEL)),
modifier,
&left_context,
&right_context,
@@ -2572,9 +2594,31 @@ impl PromPlanner {
continue;
}
let accepts_empty = matcher.is_match("");
let column_name = Self::find_case_sensitive_column(table_schema, matcher.name.as_str());
let col = if let Some(column_name) = column_name {
DfExpr::Column(Column::from_name(column_name))
let column = DfExpr::Column(Column::from_name(&column_name));
let field = table_schema
.index_of_column_by_name(None, &column_name)
.map(|index| table_schema.field(index));
if accepts_empty
&& column_name == OTLP_AGGREGATION_TEMPORALITY_LABEL
&& let Some(data_type) = field
.filter(|field| {
field.is_nullable()
&& Self::string_value_data_type(field.data_type()).is_some()
})
.map(|field| field.data_type())
{
let empty = Self::string_scalar_value(data_type, Some(String::new()))
.expect("nullable label has a string type");
DfExpr::ScalarFunction(ScalarFunction {
func: coalesce(),
args: vec![column, DfExpr::Literal(empty, None)],
})
} else {
column
}
} else {
DfExpr::Literal(ScalarValue::Utf8(Some(String::new())), None)
.alias(matcher.name.clone())
@@ -3248,6 +3292,7 @@ impl PromPlanner {
mut other_input_exprs: VecDeque<DfExpr>,
float_field: &str,
histogram_field: &str,
input_schema: &DFSchemaRef,
) -> Result<Option<Vec<DfExpr>>> {
let returns_histogram = matches!(
func.name,
@@ -3289,25 +3334,51 @@ impl PromPlanner {
});
}
let mut args = Vec::with_capacity(other_input_exprs.len() + 6);
args.push(lit(func.name));
args.push(DfExpr::Column(Column::from_name(
let timestamp_range = DfExpr::Column(Column::from_name(
RangeManipulate::build_timestamp_range_name(
self.ctx.time_index_column.as_ref().unwrap(),
),
)));
args.push(DfExpr::Column(Column::from_name(float_field)));
args.push(DfExpr::Column(Column::from_name(histogram_field)));
));
let float_range = DfExpr::Column(Column::from_name(float_field));
let histogram_range = DfExpr::Column(Column::from_name(histogram_field));
let mut args = Vec::with_capacity(other_input_exprs.len() + 6);
args.push(lit(func.name));
args.push(timestamp_range.clone());
args.push(float_range.clone());
args.push(histogram_range.clone());
args.extend(other_input_exprs);
if matches!(func.name, "rate" | "increase" | "delta") {
args.push(self.create_time_index_column_expr()?);
args.push(lit(self.ctx.range.context(ExpectRangeSelectorSnafu)?));
}
let float_expr = DfExpr::ScalarFunction(ScalarFunction {
let mut float_expr = DfExpr::ScalarFunction(ScalarFunction {
func: Arc::new(MixedRange::float_udf(self.promql_annotations.clone())),
args: args.clone(),
});
if matches!(func.name, "rate" | "increase") {
let raw_delta_function = if func.name == "rate" {
"raw_delta_rate"
} else {
"raw_delta_increase"
};
let delta_sum = DfExpr::ScalarFunction(ScalarFunction {
func: Arc::new(MixedRange::float_udf(self.promql_annotations.clone())),
args: vec![
lit(raw_delta_function),
timestamp_range,
float_range,
histogram_range,
],
});
float_expr = self.select_delta_range_math(
func.name,
input_schema,
self.ctx.range.context(ExpectRangeSelectorSnafu)?,
delta_sum,
float_expr,
)?;
}
let exprs = if returns_histogram {
self.ctx.field_columns = vec![float_field.to_string(), histogram_field.to_string()];
vec![
@@ -3348,6 +3419,7 @@ impl PromPlanner {
other_input_exprs.clone(),
&float_field,
&histogram_field,
input_schema,
)?
{
return Ok((exprs, vec![]));
@@ -3870,21 +3942,38 @@ impl PromPlanner {
let _ = other_input_exprs.remove(field_column_pos + 1);
let _ = other_input_exprs.remove(field_column_pos);
}
ScalarFunc::ExtrapolateUdf(func, range_length) => {
ScalarFunc::ExtrapolateUdf(udf, range_length) => {
let ts_range_expr = DfExpr::Column(Column::from_name(
RangeManipulate::build_timestamp_range_name(
self.ctx.time_index_column.as_ref().unwrap(),
),
));
other_input_exprs.insert(field_column_pos, ts_range_expr);
other_input_exprs.insert(field_column_pos + 1, col_expr);
other_input_exprs.insert(field_column_pos, ts_range_expr.clone());
other_input_exprs.insert(field_column_pos + 1, col_expr.clone());
other_input_exprs
.insert(field_column_pos + 2, self.create_time_index_column_expr()?);
other_input_exprs.push_back(lit(range_length));
let fn_expr = DfExpr::ScalarFunction(ScalarFunction {
func,
func: udf,
args: other_input_exprs.clone().into(),
});
let fn_expr = if matches!(func.name, "rate" | "increase")
&& !all_field_columns_are_native_histogram_ranges
{
let delta_sum = DfExpr::ScalarFunction(ScalarFunction {
func: Arc::new(SumOverTime::scalar_udf()),
args: vec![ts_range_expr, col_expr],
});
self.select_delta_range_math(
func.name,
input_schema,
range_length,
delta_sum,
fn_expr,
)?
} else {
fn_expr
};
exprs.push(fn_expr);
let _ = other_input_exprs.pop_back();
let _ = other_input_exprs.remove(field_column_pos + 2);
@@ -3917,6 +4006,49 @@ impl PromPlanner {
Ok((exprs, new_tags))
}
fn select_delta_range_math(
&self,
function: &str,
input_schema: &DFSchemaRef,
range_length: Millisecond,
delta_sum: DfExpr,
cumulative: DfExpr,
) -> Result<DfExpr> {
let marker_is_delta = if self
.ctx
.tag_columns
.iter()
.any(|tag| tag == OTLP_AGGREGATION_TEMPORALITY_LABEL)
{
Self::field_column_type(input_schema, OTLP_AGGREGATION_TEMPORALITY_LABEL)
.filter(|data_type| Self::string_value_data_type(data_type).is_some())
.map(|_| {
DfExpr::Column(Column::from_name(OTLP_AGGREGATION_TEMPORALITY_LABEL))
.eq(lit(GREPTIME_TEMPORALITY_DELTA))
})
} else {
None
};
let Some(marker_is_delta) = marker_is_delta else {
return Ok(cumulative);
};
let delta = if function == "rate" {
DfExpr::BinaryExpr(BinaryExpr {
left: Box::new(delta_sum),
op: Operator::Divide,
right: Box::new(lit(range_length as f64 / 1000.0)),
})
} else {
delta_sum
};
let display_name = cumulative.schema_name().to_string();
when(marker_is_delta, delta)
.otherwise(cumulative)
.context(DataFusionPlanningSnafu)
.map(|expr| expr.alias(display_name))
}
/// Validate label name according to Prometheus specification.
/// Label names must match the regex: [a-zA-Z_][a-zA-Z0-9_]*
/// Additionally, label names starting with double underscores are reserved for internal use.
@@ -5441,6 +5573,10 @@ impl PromPlanner {
.any(|field| field.name() == DATA_SCHEMA_TSID_COLUMN_NAME)
}
fn is_empty_metric(plan: &LogicalPlan) -> bool {
matches!(plan, LogicalPlan::Extension(Extension { node }) if node.as_any().is::<EmptyMetric>())
}
fn native_histogram_arrow_type() -> ArrowDataType {
native_histogram_value_type().as_arrow_type()
}
@@ -5706,7 +5842,7 @@ impl PromPlanner {
left_context: &PromPlannerContext,
right_context: &PromPlannerContext,
) -> Result<LogicalPlan> {
let (mut left_tag_columns, mut right_tag_columns, force_empty_join) = self
let (mut left_tag_columns, mut right_tag_columns, mut force_empty_join) = self
.binary_join_key_columns(
left.schema(),
right.schema(),
@@ -5715,6 +5851,35 @@ impl PromPlanner {
only_join_time_index,
modifier,
)?;
let use_tsid_join = !only_join_time_index
&& !force_empty_join
&& left_tag_columns == BTreeSet::from([DATA_SCHEMA_TSID_COLUMN_NAME.to_string()])
&& right_tag_columns == BTreeSet::from([DATA_SCHEMA_TSID_COLUMN_NAME.to_string()]);
let (left, right) = if !only_join_time_index
&& !use_tsid_join
&& Self::only_temporality_match_label_mismatches(left_context, right_context, modifier)
{
let mut aligned_left_context = left_context.clone();
let mut aligned_right_context = right_context.clone();
let (left, right, _) = Self::align_temporality_match_column(
left,
right,
&mut aligned_left_context,
&mut aligned_right_context,
)?;
(left_tag_columns, right_tag_columns, force_empty_join) = self
.binary_join_key_columns(
left.schema(),
right.schema(),
&aligned_left_context,
&aligned_right_context,
false,
modifier,
)?;
(left, right)
} else {
(left, right)
};
// push time index column if it exists
if let (Some(left_time_index_column), Some(right_time_index_column)) =
@@ -5755,6 +5920,145 @@ impl PromPlanner {
.context(DataFusionPlanningSnafu)
}
fn selected_binary_match_labels(
left_context: &PromPlannerContext,
right_context: &PromPlannerContext,
modifier: &Option<BinModifier>,
) -> BTreeSet<String> {
let mut labels = left_context
.tag_columns
.iter()
.chain(&right_context.tag_columns)
.cloned()
.collect::<BTreeSet<_>>();
if let Some(matching) = modifier
.as_ref()
.and_then(|modifier| modifier.matching.as_ref())
{
match matching {
LabelModifier::Include(on) => {
labels = on
.labels
.iter()
.filter(|label| {
left_context.tag_columns.contains(label)
|| right_context.tag_columns.contains(label)
})
.cloned()
.collect();
}
LabelModifier::Exclude(ignoring) => {
for label in &ignoring.labels {
labels.remove(label);
}
}
}
}
labels
}
fn only_temporality_match_label_mismatches(
left_context: &PromPlannerContext,
right_context: &PromPlannerContext,
modifier: &Option<BinModifier>,
) -> bool {
let mut mismatches =
Self::selected_binary_match_labels(left_context, right_context, modifier)
.into_iter()
.filter(|label| {
left_context.tag_columns.contains(label)
!= right_context.tag_columns.contains(label)
});
matches!(
(mismatches.next(), mismatches.next()),
(Some(label), None) if label == OTLP_AGGREGATION_TEMPORALITY_LABEL
)
}
fn align_temporality_match_column(
mut left: LogicalPlan,
mut right: LogicalPlan,
left_context: &mut PromPlannerContext,
right_context: &mut PromPlannerContext,
) -> Result<(LogicalPlan, LogicalPlan, bool)> {
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
let left_has_marker = left_context.tag_columns.iter().any(|tag| tag == marker);
let (present, add_to_left) = if left_has_marker {
(&left, false)
} else {
(&right, true)
};
let data_type = present
.schema()
.fields()
.iter()
.find(|field| field.name() == marker)
.map(|field| field.data_type().clone())
.with_context(|| ColumnNotFoundSnafu {
col: marker.to_string(),
})?;
let null = Self::string_scalar_value(&data_type, None).with_context(|| {
UnexpectedPlanExprSnafu {
desc: format!("temporality match label {marker} must be a string"),
}
})?;
let add_marker = |plan: LogicalPlan| {
let visible = plan
.schema()
.iter()
.map(|(qualifier, field)| {
DfExpr::Column(Column::new(qualifier.cloned(), field.name().clone()))
})
.collect::<Vec<_>>();
LogicalPlanBuilder::from(plan)
.project(
visible
.into_iter()
.chain([DfExpr::Literal(null, None).alias(marker)]),
)
.context(DataFusionPlanningSnafu)?
.build()
.context(DataFusionPlanningSnafu)
};
if add_to_left {
left = add_marker(left)?;
left_context.tag_columns.push(marker.to_string());
} else {
right = add_marker(right)?;
right_context.tag_columns.push(marker.to_string());
}
Ok((left, right, add_to_left))
}
fn normalized_match_key_expr(
label: &str,
field: Option<(Option<TableReference>, ArrowDataType)>,
value_type: &ArrowDataType,
internal_name: &str,
) -> DfExpr {
let empty = Self::string_scalar_value(value_type, Some(String::new()))
.expect("match label value type is a string");
let expr = if let Some((qualifier, data_type)) = field {
let column = DfExpr::Column(Column::new(qualifier, label));
let column = if &data_type == value_type {
column
} else {
DfExpr::Cast(Cast {
expr: Box::new(column),
data_type: value_type.clone(),
})
};
DfExpr::ScalarFunction(ScalarFunction {
func: coalesce(),
args: vec![column, DfExpr::Literal(empty, None)],
})
} else {
DfExpr::Literal(empty, None)
};
expr.alias(internal_name)
}
fn is_zero_row_empty_relation(plan: &LogicalPlan) -> bool {
// `produce_one_row` is used for input-free plans that still emit one row;
// only the false case is a statically proven empty vector.
@@ -5764,19 +6068,19 @@ impl PromPlanner {
/// Build a set operator (AND/OR/UNLESS)
fn set_op_on_non_field_columns(
&mut self,
left: LogicalPlan,
mut left: LogicalPlan,
mut right: LogicalPlan,
left_context: PromPlannerContext,
right_context: PromPlannerContext,
op: TokenType,
modifier: &Option<BinModifier>,
) -> Result<LogicalPlan> {
let mut left_tag_col_set = left_context
let left_tag_col_set = left_context
.tag_columns
.iter()
.cloned()
.collect::<HashSet<_>>();
let mut right_tag_col_set = right_context
let right_tag_col_set = right_context
.tag_columns
.iter()
.cloned()
@@ -5794,9 +6098,7 @@ impl PromPlanner {
);
}
// apply modifier
if let Some(modifier) = modifier {
// one-to-many and many-to-one are not supported
ensure!(
matches!(
modifier.card,
@@ -5806,44 +6108,68 @@ impl PromPlanner {
name: modifier.card.clone(),
},
);
// apply label modifier
if let Some(matching) = &modifier.matching {
match matching {
// keeps columns mentioned in `on`
LabelModifier::Include(on) => {
let mask = on.labels.iter().cloned().collect::<HashSet<_>>();
left_tag_col_set = left_tag_col_set.intersection(&mask).cloned().collect();
right_tag_col_set =
right_tag_col_set.intersection(&mask).cloned().collect();
}
// removes columns memtioned in `ignoring`
LabelModifier::Exclude(ignoring) => {
// doesn't check existence of label
for label in &ignoring.labels {
let _ = left_tag_col_set.remove(label);
let _ = right_tag_col_set.remove(label);
}
}
let output_context = left_context.clone();
let visible_left_schema = left.schema().clone();
let mut left_context = left_context;
let mut right_context = right_context;
let added_marker_to_left = if Self::only_temporality_match_label_mismatches(
&left_context,
&right_context,
modifier,
) {
let aligned = Self::align_temporality_match_column(
left,
right,
&mut left_context,
&mut right_context,
)?;
left = aligned.0;
right = aligned.1;
aligned.2
} else {
false
};
let mut left_tag_col_set = left_context
.tag_columns
.iter()
.cloned()
.collect::<BTreeSet<_>>();
let mut right_tag_col_set = right_context
.tag_columns
.iter()
.cloned()
.collect::<BTreeSet<_>>();
if let Some(matching) = modifier
.as_ref()
.and_then(|modifier| modifier.matching.as_ref())
{
match matching {
LabelModifier::Include(on) => {
let mask = on.labels.iter().cloned().collect::<BTreeSet<_>>();
left_tag_col_set = left_tag_col_set.intersection(&mask).cloned().collect();
right_tag_col_set = right_tag_col_set.intersection(&mask).cloned().collect();
}
LabelModifier::Exclude(ignoring) => {
for label in &ignoring.labels {
let _ = left_tag_col_set.remove(label);
let _ = right_tag_col_set.remove(label);
}
}
}
}
// ensure two sides have the same tag columns
if !matches!(op.id(), token::T_LOR) {
ensure!(
left_tag_col_set == right_tag_col_set,
CombineTableColumnMismatchSnafu {
left: left_tag_col_set.into_iter().collect::<Vec<_>>(),
right: right_tag_col_set.into_iter().collect::<Vec<_>>(),
}
)
};
ensure!(
left_tag_col_set == right_tag_col_set,
CombineTableColumnMismatchSnafu {
left: left_tag_col_set.iter().cloned().collect::<Vec<_>>(),
right: right_tag_col_set.iter().cloned().collect::<Vec<_>>(),
}
);
let left_time_index = left_context.time_index_column.clone().unwrap();
let right_time_index = right_context.time_index_column.clone().unwrap();
let join_keys = left_tag_col_set
.iter()
.cloned()
.chain([left_time_index.clone()])
.collect::<Vec<_>>();
// alias right time index column if necessary
if left_context.time_index_column != right_context.time_index_column {
@@ -5867,6 +6193,11 @@ impl PromPlanner {
.context(DataFusionPlanningSnafu)?;
}
let join_keys = left_tag_col_set
.into_iter()
.chain([left_time_index])
.collect::<Vec<_>>();
ensure!(
left_context.field_columns.len() == 1
|| Self::field_columns_are_alternative_samples(
@@ -5913,9 +6244,20 @@ impl PromPlanner {
}
_ => UnexpectedTokenSnafu { token: op }.fail(),
}?;
let result = if added_marker_to_left {
LogicalPlanBuilder::from(result)
.project(visible_left_schema.iter().map(|(qualifier, field)| {
DfExpr::Column(Column::new(qualifier.cloned(), field.name().clone()))
}))
.context(DataFusionPlanningSnafu)?
.build()
.context(DataFusionPlanningSnafu)?
} else {
result
};
// AND/UNLESS preserve the complete left operand schema and metadata.
self.ctx = left_context;
self.ctx = output_context;
Ok(result)
}
@@ -6492,8 +6834,6 @@ impl PromPlanner {
}
.fail();
};
let empty = Self::string_scalar_value(&value_type, Some(String::new()))
.expect("match label value type is a string");
let internal_name = loop {
let name = format!("__promql_or_match_{next_internal_column}");
next_internal_column += 1;
@@ -6501,28 +6841,18 @@ impl PromPlanner {
break name;
}
};
let normalize = |field: Option<(Option<TableReference>, ArrowDataType)>| {
let expr = if let Some((qualifier, data_type)) = field {
let column = DfExpr::Column(Column::new(qualifier, label.clone()));
let column = if data_type == value_type {
column
} else {
DfExpr::Cast(Cast {
expr: Box::new(column),
data_type: value_type.clone(),
})
};
DfExpr::ScalarFunction(ScalarFunction {
func: coalesce(),
args: vec![column, DfExpr::Literal(empty.clone(), None)],
})
} else {
DfExpr::Literal(empty.clone(), None)
};
expr.alias(internal_name.clone())
};
left_match_exprs.push(normalize(left_field));
right_match_exprs.push(normalize(right_field));
left_match_exprs.push(Self::normalized_match_key_expr(
label,
left_field,
&value_type,
&internal_name,
));
right_match_exprs.push(Self::normalized_match_key_expr(
label,
right_field,
&value_type,
&internal_name,
));
}
let left_augmented = LogicalPlanBuilder::from(left_projected)
@@ -6850,6 +7180,8 @@ mod test {
use crate::parser::QueryLanguageParser;
use crate::query_engine::DefaultSerializer;
mod delta;
fn find_instant_manipulate(plan: &LogicalPlan) -> Option<&InstantManipulate> {
if let LogicalPlan::Extension(Extension { node }) = plan
&& let Some(instant_manipulate) = node.as_any().downcast_ref::<InstantManipulate>()
@@ -7757,8 +8089,15 @@ mod test {
}
async fn build_test_native_histogram_table_provider(table_name: &str) -> DfTableSourceProvider {
build_test_native_histogram_table_provider_with_marker(table_name, false).await
}
async fn build_test_native_histogram_table_provider_with_marker(
table_name: &str,
temporality_marker: bool,
) -> DfTableSourceProvider {
let catalog_list = MemoryCatalogManager::with_default_setup();
let columns = vec![
let mut columns = vec![
ColumnSchema::new(
"tag_0".to_string(),
ConcreteDataType::string_datatype(),
@@ -7769,6 +8108,16 @@ mod test {
ConcreteDataType::string_datatype(),
true,
),
];
if temporality_marker {
columns.push(ColumnSchema::new(
OTLP_AGGREGATION_TEMPORALITY_LABEL.to_string(),
ConcreteDataType::string_datatype(),
true,
));
}
let tag_count = columns.len();
columns.extend([
ColumnSchema::new(
"timestamp".to_string(),
ConcreteDataType::timestamp_millisecond_datatype(),
@@ -7780,12 +8129,12 @@ mod test {
native_histogram_value_type().clone(),
true,
),
];
]);
let schema = Arc::new(Schema::new(columns));
let table_meta = TableMetaBuilder::empty()
.schema(schema)
.primary_key_indices(vec![0, 1])
.value_indices(vec![3])
.primary_key_indices((0..tag_count).collect())
.value_indices(vec![tag_count + 1])
.next_column_id(1024)
.build()
.unwrap();
@@ -7880,14 +8229,29 @@ mod test {
async fn build_test_mixed_native_histogram_table_provider(
table_name: &str,
) -> DfTableSourceProvider {
build_test_mixed_native_histogram_table_provider_with_marker(table_name, false).await
}
async fn build_test_mixed_native_histogram_table_provider_with_marker(
table_name: &str,
temporality_marker: bool,
) -> DfTableSourceProvider {
let catalog_list = MemoryCatalogManager::with_default_setup();
let columns = vec![
ColumnSchema::new(
"tag_0".to_string(),
let mut columns = vec![ColumnSchema::new(
"tag_0".to_string(),
ConcreteDataType::string_datatype(),
false,
)];
if temporality_marker {
columns.push(ColumnSchema::new(
OTLP_AGGREGATION_TEMPORALITY_LABEL.to_string(),
ConcreteDataType::string_datatype(),
false,
),
true,
));
}
let tag_count = columns.len();
columns.extend([
ColumnSchema::new(
"timestamp".to_string(),
ConcreteDataType::timestamp_millisecond_datatype(),
@@ -7904,12 +8268,12 @@ mod test {
ConcreteDataType::float64_datatype(),
true,
),
];
]);
let schema = Arc::new(Schema::new(columns));
let table_meta = TableMetaBuilder::empty()
.schema(schema.clone())
.primary_key_indices(vec![0])
.value_indices(vec![2, 3])
.primary_key_indices((0..tag_count).collect())
.value_indices(vec![tag_count + 1, tag_count + 2])
.next_column_id(1024)
.build()
.unwrap();
@@ -7920,16 +8284,20 @@ mod test {
.build()
.unwrap(),
);
let batch = RecordBatch::try_new(
schema.arrow_schema().clone(),
vec![
Arc::new(StringArray::from(vec!["float", "histogram"])),
Arc::new(TimestampMillisecondArray::from(vec![1_000, 1_000])),
build_histogram_array(&[None, Some(direct_or_histogram())]),
Arc::new(Float64Array::from(vec![Some(2.0), None])),
],
)
.unwrap();
let mut arrays: Vec<Arc<dyn Array>> =
vec![Arc::new(StringArray::from(vec!["float", "histogram"]))];
if temporality_marker {
arrays.push(Arc::new(StringArray::from(vec![
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
])));
}
arrays.extend([
Arc::new(TimestampMillisecondArray::from(vec![1_000, 1_000])) as Arc<dyn Array>,
build_histogram_array(&[None, Some(direct_or_histogram())]),
Arc::new(Float64Array::from(vec![Some(2.0), None])),
]);
let batch = RecordBatch::try_new(schema.arrow_schema().clone(), arrays).unwrap();
let backing = GreptimeMemTable::new_with_catalog(
table_name,
GreptimeRecordBatch::from_df_record_batch(schema, batch),
src/query/src/promql/planner/test/delta.rs
+759
View File
@@ -0,0 +1,759 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use common_query::logical_plan::SubstraitPlanDecoder;
use common_query::prelude::set_default_prefix;
use datafusion::catalog::SchemaProvider;
use super::*;
use crate::query_engine::DefaultPlanDecoder;
fn delta_temporality_table_provider() -> (DfTableSourceProvider, QueryEngineState, Arc<MemTable>) {
let catalog = MemoryCatalogManager::with_default_setup();
let schema = Arc::new(Schema::new(vec![
ColumnSchema::new(
"series".to_string(),
ConcreteDataType::string_datatype(),
false,
),
ColumnSchema::new(
OTLP_AGGREGATION_TEMPORALITY_LABEL.to_string(),
ConcreteDataType::string_datatype(),
true,
),
ColumnSchema::new(
greptime_timestamp().to_string(),
ConcreteDataType::timestamp_millisecond_datatype(),
false,
)
.with_time_index(true),
ColumnSchema::new(
greptime_value().to_string(),
ConcreteDataType::float64_datatype(),
true,
),
]));
let batch = RecordBatch::try_new(
schema.arrow_schema().clone(),
vec![
Arc::new(StringArray::from(vec![
"delta",
"delta",
"delta",
"single",
"stale",
"stale",
"cumulative",
"cumulative",
"cumulative",
])),
Arc::new(StringArray::from(vec![
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
Some(GREPTIME_TEMPORALITY_DELTA),
None,
None,
None,
])),
Arc::new(TimestampMillisecondArray::from(vec![
60_000, 120_000, 180_000, 180_000, 120_000, 180_000, 60_000, 120_000, 180_000,
])),
Arc::new(Float64Array::from(vec![
10.0,
20.0,
15.0,
7.0,
7.0,
f64::from_bits(PROMETHEUS_STALE_NAN_BITS),
10.0,
20.0,
30.0,
])),
],
)
.unwrap();
let datafusion_table =
Arc::new(MemTable::try_new(batch.schema(), vec![vec![batch.clone()]]).unwrap());
let table_meta = TableMetaBuilder::empty()
.schema(schema.clone())
.primary_key_indices(vec![0, 1])
.value_indices(vec![3])
.next_column_id(4)
.build()
.unwrap();
let table_info = Arc::new(
TableInfoBuilder::default()
.name("delta_metric")
.meta(table_meta)
.build()
.unwrap(),
);
let backing = GreptimeMemTable::new_with_catalog(
"delta_metric",
GreptimeRecordBatch::from_df_record_batch(schema, batch),
4_001,
DEFAULT_CATALOG_NAME.to_string(),
DEFAULT_SCHEMA_NAME.to_string(),
);
let table = Arc::new(Table::new(
table_info,
FilterPushDownType::Unsupported,
backing.data_source(),
));
catalog
.register_table_sync(RegisterTableRequest {
catalog: DEFAULT_CATALOG_NAME.to_string(),
schema: DEFAULT_SCHEMA_NAME.to_string(),
table_name: "delta_metric".to_string(),
table_id: 4_001,
table,
})
.unwrap();
let state = QueryEngineState::new(
catalog.clone(),
None,
None,
None,
None,
None,
false,
Plugins::default(),
QueryOptions::default(),
);
let provider = DfTableSourceProvider::new(
catalog,
false,
QueryContext::arc(),
DummyDecoder::arc(),
false,
);
(provider, state, datafusion_table)
}
#[tokio::test]
async fn rate_and_increase_select_raw_delta_math_per_series() {
set_default_prefix(Some("custom")).unwrap();
assert_eq!(
OTLP_AGGREGATION_TEMPORALITY_LABEL,
"otlp_aggregation_temporality"
);
let eval_time = UNIX_EPOCH.checked_add(Duration::from_secs(180)).unwrap();
for (function, expected_delta, expected_single) in
[("increase", 45.0, 7.0), ("rate", 0.25, 7.0 / 180.0)]
{
let eval_stmt = EvalStmt {
expr: parser::parse(&format!("{function}(delta_metric[3m])")).unwrap(),
start: eval_time,
end: eval_time,
interval: Duration::from_secs(60),
lookback_delta: Duration::from_secs(300),
};
let (provider, state, datafusion_table) = delta_temporality_table_provider();
let raw = PromPlanner::stmt_to_plan(provider, &eval_stmt, &state)
.await
.unwrap();
let plan = raw.display_indent_schema().to_string();
assert!(plan.contains("prom_sum_over_time"), "{plan}");
assert!(plan.contains(OTLP_AGGREGATION_TEMPORALITY_LABEL), "{plan}");
let value_field = raw
.schema()
.fields()
.iter()
.find(|field| field.data_type() == &ArrowDataType::Float64)
.unwrap()
.name()
.clone();
assert!(value_field.starts_with(&format!("prom_{function}")));
let executable = if function == "increase" {
let context = SessionContext::new_with_state(state.session_state());
let catalog = Arc::new(MemoryCatalogProvider::new());
let schema = Arc::new(MemorySchemaProvider::new());
schema
.register_table("delta_metric".to_string(), datafusion_table)
.unwrap();
catalog
.register_schema(DEFAULT_SCHEMA_NAME, schema)
.unwrap();
context.register_catalog("datafusion", catalog);
let decoder = DefaultPlanDecoder::new(context.state(), &QueryContext::arc()).unwrap();
decoder
.decode(
DFLogicalSubstraitConvertor
.encode(&raw, DefaultSerializer)
.unwrap(),
context.state().catalog_list().clone(),
false,
)
.await
.unwrap()
} else {
raw
};
let (_, batches) = execute(executable, &state).await;
let mut results = HashMap::new();
for batch in batches {
let series = batch
.column_by_name("series")
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
let temporality = batch
.column_by_name(OTLP_AGGREGATION_TEMPORALITY_LABEL)
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
let values = batch
.column_by_name(&value_field)
.unwrap()
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
for row in 0..batch.num_rows() {
results.insert(
series.value(row).to_string(),
(
(!temporality.is_null(row)).then(|| temporality.value(row).to_string()),
values.value(row),
),
);
}
}
assert_eq!(
Some(&(Some(GREPTIME_TEMPORALITY_DELTA.to_string()), expected_delta)),
results.get("delta")
);
assert_eq!(
Some(&(
Some(GREPTIME_TEMPORALITY_DELTA.to_string()),
expected_single
)),
results.get("single")
);
assert_eq!(
Some(&(
Some(GREPTIME_TEMPORALITY_DELTA.to_string()),
expected_single
)),
results.get("stale")
);
assert!(results.contains_key("cumulative"));
}
}
#[tokio::test]
async fn empty_metric_broadcasts_over_temporality_marker() {
let eval_time = UNIX_EPOCH.checked_add(Duration::from_secs(180)).unwrap();
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
for query in [
"vector(2) * delta_metric".to_string(),
format!("vector(2) * ignoring({marker}) delta_metric"),
"delta_metric * vector(2)".to_string(),
format!("delta_metric * ignoring({marker}) vector(2)"),
] {
let eval_stmt = EvalStmt {
expr: parser::parse(&query).unwrap(),
start: eval_time,
end: eval_time,
interval: Duration::from_secs(60),
lookback_delta: Duration::from_secs(300),
};
let (provider, state, _) = delta_temporality_table_provider();
let plan = PromPlanner::stmt_to_plan(provider, &eval_stmt, &state)
.await
.unwrap();
let plan_text = plan.display_indent_schema().to_string();
assert!(
!plan_text.contains("Filter: Boolean(false)"),
"{query}: {plan_text}"
);
let value_field = plan
.schema()
.fields()
.iter()
.find(|field| field.data_type() == &ArrowDataType::Float64)
.unwrap()
.name()
.clone();
let (_, batches) = execute(plan, &state).await;
let mut results = HashMap::new();
for batch in batches {
let series = batch
.column_by_name("series")
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
let values = batch
.column_by_name(&value_field)
.unwrap()
.as_any()
.downcast_ref::<Float64Array>()
.unwrap();
let temporality = batch
.column_by_name(marker)
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
for row in 0..batch.num_rows() {
results.insert(
series.value(row).to_string(),
(
(!temporality.is_null(row)).then(|| temporality.value(row).to_string()),
values.value(row),
),
);
}
}
assert_eq!(3, results.len(), "{query}");
assert_eq!(
Some(&(Some(GREPTIME_TEMPORALITY_DELTA.to_string()), 30.0)),
results.get("delta"),
"{query}"
);
assert_eq!(
Some(&(Some(GREPTIME_TEMPORALITY_DELTA.to_string()), 14.0)),
results.get("single"),
"{query}"
);
assert_eq!(Some(&(None, 60.0)), results.get("cumulative"), "{query}");
}
}
#[tokio::test]
async fn native_histogram_rate_ignores_delta_marker() {
let provider =
build_test_native_histogram_table_provider_with_marker("native_delta_metric", true).await;
let plan = PromPlanner::stmt_to_plan(
provider,
&build_eval_stmt("rate(native_delta_metric[5m])"),
&build_query_engine_state(),
)
.await
.unwrap()
.display_indent_schema()
.to_string();
assert!(plan.contains("prom_native_histogram_rate"), "{plan}");
assert!(!plan.contains("prom_sum_over_time"), "{plan}");
assert!(!plan.contains("CASE WHEN"), "{plan}");
}
#[tokio::test]
async fn mixed_range_rate_selects_delta_math_only_for_float_samples() {
let plan = PromPlanner::stmt_to_plan(
build_test_mixed_native_histogram_table_provider_with_marker("some_metric", true).await,
&build_eval_stmt("rate(some_metric[5m])"),
&build_query_engine_state(),
)
.await
.unwrap()
.display_indent_schema()
.to_string();
assert!(plan.contains("CASE WHEN"), "{plan}");
assert!(plan.contains("prom_mixed_range_float"), "{plan}");
assert!(plan.contains("prom_mixed_range_histogram"), "{plan}");
assert!(plan.contains(OTLP_AGGREGATION_TEMPORALITY_LABEL), "{plan}");
}
#[tokio::test]
async fn delta_mixed_ranges_drop_and_float_ranges_sum() {
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
for function in ["rate", "increase"] {
for histogram_present in [true, false] {
let float = source(
"lhs",
false,
1_000,
vec![
("job", Some("job")),
(marker, Some(GREPTIME_TEMPORALITY_DELTA)),
],
DirectOrValue::Float64(1.0),
);
let histogram = source(
"rhs",
!histogram_present,
2_000,
vec![
("job", Some("job")),
(marker, Some(GREPTIME_TEMPORALITY_DELTA)),
],
DirectOrValue::NativeHistogram(direct_or_histogram()),
);
let collector = PromqlAnnotationCollector::default();
let mut planner = PromPlanner {
table_provider: build_test_table_provider_with_fields(
&[(DEFAULT_SCHEMA_NAME.to_string(), "dummy".to_string())],
&[],
)
.await,
ctx: PromPlannerContext::default(),
promql_annotations: Some(collector.clone()),
};
let left_context = direct_or_context("lhs", &["job", marker], "v");
let right_context = direct_or_context("rhs", &["job", marker], "v");
let input = planner
.or_operator(
scan(&float),
scan(&histogram),
left_context.tag_columns.iter().cloned().collect(),
right_context.tag_columns.iter().cloned().collect(),
left_context,
right_context,
&or_modifier("lhs or rhs"),
)
.unwrap();
let mut sort_exprs = planner
.ctx
.tag_columns
.iter()
.map(|tag| DfExpr::Column(Column::from_name(tag)).sort(true, true))
.collect_vec();
sort_exprs.push(DfExpr::Column(Column::from_name("ts")).sort(true, true));
let input = LogicalPlanBuilder::from(input)
.sort(sort_exprs)
.unwrap()
.build()
.unwrap();
let input = LogicalPlan::Extension(Extension {
node: Arc::new(SeriesDivide::new(
planner.ctx.tag_columns.clone(),
"ts".to_string(),
input,
)),
});
planner.ctx.start = 2_000;
planner.ctx.end = 2_000;
planner.ctx.interval = 1_000;
planner.ctx.range = Some(2_000);
let input = LogicalPlan::Extension(Extension {
node: Arc::new(
RangeManipulate::new(
2_000,
2_000,
1_000,
2_000,
"ts".to_string(),
planner.ctx.field_columns.clone(),
input,
)
.unwrap(),
),
});
let PromExpr::Call(call) = parser::parse(&format!("{function}(mixed[2s])")).unwrap()
else {
unreachable!()
};
let preserve_any_value = PromPlanner::field_columns_are_alternative_samples(
input.schema(),
&planner.ctx.field_columns,
);
let state = build_query_engine_state();
let (mut exprs, _) = planner
.create_function_expr(&call.func, vec![], input.schema(), &state)
.unwrap();
exprs.insert(0, planner.create_time_index_column_expr().unwrap());
let plan = LogicalPlanBuilder::from(input)
.project(exprs)
.unwrap()
.filter(
planner
.create_empty_values_filter_expr(preserve_any_value)
.unwrap(),
)
.unwrap()
.build()
.unwrap();
let value_field = plan
.schema()
.fields()
.iter()
.find(|field| field.data_type() == &ArrowDataType::Float64)
.unwrap()
.name()
.clone();
let (_, batches) = execute(plan, &state).await;
let row_count = batches.iter().map(RecordBatch::num_rows).sum::<usize>();
if histogram_present {
assert_eq!(0, row_count, "{function}");
} else {
assert_eq!(1, row_count, "{function}");
let expected = if function == "rate" { 0.5 } else { 1.0 };
assert_eq!(vec![expected], values(&batches, &value_field), "{function}");
}
let mut warnings = Vec::new();
let mut infos = Vec::new();
collector.append_to(&mut warnings, &mut infos);
let expected_warnings = if histogram_present {
vec![format!(
"{function}: encountered a mix of float and native histogram samples"
)]
} else {
vec![]
};
assert_eq!(expected_warnings, warnings);
assert!(infos.is_empty(), "{function}: {infos:?}");
}
}
}
#[tokio::test]
async fn temporality_matchers_treat_null_as_absent() {
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
let schema = Arc::new(ArrowSchema::new(vec![Field::new(
marker,
ArrowDataType::Utf8,
true,
)]));
let batch = RecordBatch::try_new(
schema.clone(),
vec![Arc::new(StringArray::from(vec![
None,
Some(GREPTIME_TEMPORALITY_DELTA),
]))],
)
.unwrap();
let table = Arc::new(MemTable::try_new(schema, vec![vec![batch]]).unwrap());
for (matcher, expected) in [
(r#"="""#, vec![None]),
(r#"!="delta""#, vec![None]),
(r#"=~".*""#, vec![None, Some(GREPTIME_TEMPORALITY_DELTA)]),
(r#"!~"delta""#, vec![None]),
(r#"="delta""#, vec![Some(GREPTIME_TEMPORALITY_DELTA)]),
] {
let query = format!(r#"metric{{{marker}{matcher}}}"#);
let scan = LogicalPlanBuilder::scan("labels", provider_as_source(table.clone()), None)
.unwrap()
.build()
.unwrap();
let PromExpr::VectorSelector(selector) = parser::parse(&query).unwrap() else {
unreachable!()
};
let expressions = PromPlanner::matchers_to_expr(selector.matchers, scan.schema()).unwrap();
let display = expressions.iter().map(ToString::to_string).join(" AND ");
if matcher == r#"="delta""# {
assert!(!display.contains("coalesce"), "{display}");
} else if !expressions.is_empty() {
assert!(display.contains("coalesce"), "{display}");
}
let plan = if let Some(filter) = conjunction(expressions) {
LogicalPlanBuilder::from(scan)
.filter(filter)
.unwrap()
.build()
.unwrap()
} else {
scan
};
let (_, batches) = execute(plan, &build_query_engine_state()).await;
let actual = batches
.iter()
.flat_map(|batch| {
let labels = batch
.column_by_name(marker)
.unwrap()
.as_any()
.downcast_ref::<StringArray>()
.unwrap();
(0..batch.num_rows())
.map(move |row| (!labels.is_null(row)).then(|| labels.value(row).to_string()))
})
.collect::<Vec<_>>();
assert_eq!(
expected
.into_iter()
.map(|value| value.map(str::to_string))
.collect::<Vec<_>>(),
actual,
"{query}"
);
}
let ordinary_schema = Arc::new(ArrowSchema::new(vec![Field::new(
"label",
ArrowDataType::Utf8,
true,
)]));
let ordinary_scan = LogicalPlanBuilder::scan(
"ordinary_labels",
provider_as_source(Arc::new(
MemTable::try_new(ordinary_schema, vec![vec![]]).unwrap(),
)),
None,
)
.unwrap()
.build()
.unwrap();
let PromExpr::VectorSelector(selector) = parser::parse(r#"metric{label!="delta"}"#).unwrap()
else {
unreachable!()
};
let expressions = PromPlanner::matchers_to_expr(selector.matchers, ordinary_scan.schema())
.unwrap()
.iter()
.map(ToString::to_string)
.join(" AND ");
assert!(!expressions.contains("coalesce"), "{expressions}");
}
#[tokio::test]
async fn binary_joins_align_only_the_temporality_marker() {
let marker = OTLP_AGGREGATION_TEMPORALITY_LABEL;
for (left_marker, expected_rows) in [(Some(GREPTIME_TEMPORALITY_DELTA), 0), (None, 1)] {
let left = source(
"lhs",
false,
1,
vec![("job", Some("job")), (marker, left_marker)],
DirectOrValue::Float64(1.0),
);
let right = source(
"rhs",
false,
1,
vec![("job", Some("job"))],
DirectOrValue::Float64(2.0),
);
let left_context = direct_or_context("lhs", &["job", marker], "v");
let right_context = direct_or_context("rhs", &["job"], "v");
let planner = PromPlanner {
table_provider: build_test_table_provider_with_fields(
&[(DEFAULT_SCHEMA_NAME.to_string(), "dummy".to_string())],
&[],
)
.await,
ctx: PromPlannerContext::default(),
promql_annotations: None,
};
let joined = planner
.join_on_non_field_columns(
scan(&left),
scan(&right),
TableReference::bare("lhs"),
TableReference::bare("rhs"),
Some("ts".to_string()),
Some("ts".to_string()),
false,
&None,
&left_context,
&right_context,
)
.unwrap();
assert!(
!joined
.display_indent_schema()
.to_string()
.contains("__promql_match_"),
"{joined:?}"
);
let (_, batches) = execute(joined, &build_query_engine_state()).await;
assert_eq!(
expected_rows,
batches.iter().map(RecordBatch::num_rows).sum::<usize>()
);
let PromExpr::Binary(and_expr) = parser::parse("lhs and rhs").unwrap() else {
unreachable!()
};
let mut planner = PromPlanner {
table_provider: build_test_table_provider_with_fields(
&[(DEFAULT_SCHEMA_NAME.to_string(), "dummy".to_string())],
&[],
)
.await,
ctx: PromPlannerContext::default(),
promql_annotations: None,
};
let set = planner
.set_op_on_non_field_columns(
scan(&left),
scan(&right),
left_context,
right_context,
and_expr.op,
&and_expr.modifier,
)
.unwrap();
assert!(
set.schema()
.fields()
.iter()
.all(|field| !field.name().starts_with("__promql_match_"))
);
assert!(
!set.display_indent_schema()
.to_string()
.contains("__promql_match_"),
"{set:?}"
);
let (_, batches) = execute(set, &build_query_engine_state()).await;
assert_eq!(
expected_rows,
batches.iter().map(RecordBatch::num_rows).sum::<usize>()
);
}
let left = source(
"lhs",
false,
1,
vec![("job", Some("job"))],
DirectOrValue::Float64(1.0),
);
let right = source(
"rhs",
false,
1,
vec![("job", Some("job")), (marker, None)],
DirectOrValue::Float64(2.0),
);
let PromExpr::Binary(and_expr) = parser::parse("lhs and rhs").unwrap() else {
unreachable!()
};
let mut planner = PromPlanner {
table_provider: build_test_table_provider_with_fields(
&[(DEFAULT_SCHEMA_NAME.to_string(), "dummy".to_string())],
&[],
)
.await,
ctx: PromPlannerContext::default(),
promql_annotations: None,
};
let set = planner
.set_op_on_non_field_columns(
scan(&left),
scan(&right),
direct_or_context("lhs", &["job"], "v"),
direct_or_context("rhs", &["job", marker], "v"),
and_expr.op,
&and_expr.modifier,
)
.unwrap();
assert!(set.schema().field_with_unqualified_name(marker).is_err());
let (_, batches) = execute(set, &build_query_engine_state()).await;
assert_eq!(1, batches.iter().map(RecordBatch::num_rows).sum::<usize>());
}
src/servers/src/error.rs
+4
View File
@@ -177,6 +177,9 @@ pub enum Error {
location: Location,
},
#[snafu(display("Invalid OTLP metric input: {}", reason))]
InvalidOtlpMetricInput { reason: String },
#[snafu(display(
"Too many concurrent large requests, limit: {}, request size: {}",
ReadableSize(*limit as u64),
@@ -755,6 +758,7 @@ impl ErrorExt for Error {
NotSupported { .. }
| InvalidParameter { .. }
| InvalidOtlpMetricInput { .. }
| InvalidQuery { .. }
| InfluxdbLineProtocol { .. }
| InvalidOpentsdbJsonRequest { .. }
src/servers/src/http/otlp.rs
+20 -8
View File
@@ -126,15 +126,24 @@ pub async fn metrics(
}));
let query_ctx = Arc::new(query_ctx);
handler.metrics(request, query_ctx).await.map(|outcome| {
if outcome.accepted_data_points == 0 && outcome.rejected_data_points > 0 {
OtlpMetricsResponse::Failure(outcome)
} else if outcome.rejected_data_points > 0 || outcome.error_message.is_some() {
OtlpMetricsResponse::PartialSuccess(outcome)
} else {
OtlpMetricsResponse::FullSuccess(outcome)
match handler.metrics(request, query_ctx).await {
Ok(outcome) => {
if outcome.accepted_data_points == 0 && outcome.rejected_data_points > 0 {
Ok(OtlpMetricsResponse::Failure(outcome))
} else if outcome.rejected_data_points > 0 || outcome.error_message.is_some() {
Ok(OtlpMetricsResponse::PartialSuccess(outcome))
} else {
Ok(OtlpMetricsResponse::FullSuccess(outcome))
}
}
})
Err(error::Error::InvalidOtlpMetricInput { reason }) => {
Ok(OtlpMetricsResponse::Failure(MetricsIngestOutcome {
error_message: Some(reason),
..Default::default()
}))
}
Err(error) => Err(error),
}
}
#[axum_macros::debug_handler]
@@ -365,3 +374,6 @@ impl IntoResponse for OtlpTraceResponse {
}
}
}
#[cfg(test)]
mod tests;
src/servers/src/http/otlp/tests.rs
+36
View File
@@ -0,0 +1,36 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use axum::body::to_bytes;
use super::*;
#[tokio::test]
async fn metric_failure_is_a_protobuf_invalid_argument() {
let response = OtlpMetricsResponse::Failure(MetricsIngestOutcome {
error_message: Some("reserved temporality label".to_string()),
..Default::default()
})
.into_response();
assert_eq!(StatusCode::BAD_REQUEST, response.status());
assert_eq!(
&CONTENT_TYPE_PROTOBUF,
response.headers().get(header::CONTENT_TYPE).unwrap()
);
let body = to_bytes(response.into_body(), 1024).await.unwrap();
let status = GoogleRpcStatus::decode(body).unwrap();
assert_eq!(tonic::Code::InvalidArgument as i32, status.code);
assert_eq!("reserved temporality label", status.message);
}
src/servers/src/http/prometheus.rs
+13 -2
View File
@@ -66,7 +66,10 @@ use store_api::metric_engine_consts::{
};
use table::TableRef;
use table::metadata::TableInfo;
use table::requests::{SEMANTIC_METRIC_TEMPORALITY, SEMANTIC_METRIC_TYPE, SEMANTIC_METRIC_UNIT};
use table::requests::{
METRIC_TEMPORALITY_DELTA, SEMANTIC_METRIC_TEMPORALITY, SEMANTIC_METRIC_TYPE,
SEMANTIC_METRIC_UNIT, SEMANTIC_VALUE_MIXED,
};
pub use super::result::prometheus_resp::PrometheusJsonResponse;
use crate::error::{
@@ -2024,7 +2027,12 @@ fn prometheus_metadata_from_table(table_info: &TableInfo) -> PromMetadata {
Some(metric_type)
if options
.get(SEMANTIC_METRIC_TEMPORALITY)
.is_some_and(|temporality| temporality == "delta")
.is_some_and(|temporality| {
matches!(
temporality.as_str(),
METRIC_TEMPORALITY_DELTA | SEMANTIC_VALUE_MIXED
)
})
&& matches!(
metric_type.as_str(),
"counter" | "histogram" | "updown_counter"
@@ -3504,6 +3512,9 @@ mod tests {
("mixed", None, "unknown"),
("counter", Some("delta"), "unknown"),
("histogram", Some("delta"), "unknown"),
("counter", Some("mixed"), "unknown"),
("histogram", Some("mixed"), "unknown"),
("updown_counter", Some("mixed"), "unknown"),
] {
let mut table_info = table_info.clone();
table_info
src/servers/src/otel_arrow.rs
+10
View File
@@ -146,6 +146,16 @@ impl ArrowMetricsService for OtelArrowServiceHandler<OpenTelemetryProtocolHandle
});
let outcome = match handler.metrics(request, query_ctx.clone()).await {
Ok(outcome) => outcome,
Err(error::Error::InvalidOtlpMetricInput { reason }) => {
let _ = sender
.send(Ok(BatchStatus {
batch_id,
status_code: ArrowStatusCode::InvalidArgument as i32,
status_message: reason,
}))
.await;
continue;
}
Err(e) => {
let _ = sender
.send(Err(Status::new(
src/servers/src/otlp/metrics.rs
+213 -69
View File
@@ -23,7 +23,10 @@ use common_grpc::precision::Precision;
use common_query::native_histogram::{
encode_native_histogram, native_histogram_column_schema, native_histogram_value_type,
};
use common_query::prelude::{GREPTIME_COUNT, greptime_timestamp, greptime_value};
use common_query::prelude::{
GREPTIME_COUNT, GREPTIME_TEMPORALITY_DELTA, OTLP_AGGREGATION_TEMPORALITY_LABEL,
greptime_timestamp, greptime_value,
};
use common_query::prometheus::PROMETHEUS_STALE_NAN_BITS;
use common_telemetry::warn;
use lazy_static::lazy_static;
@@ -32,8 +35,9 @@ use otel_arrow_rust::proto::opentelemetry::common::v1::{AnyValue, KeyValue, any_
use otel_arrow_rust::proto::opentelemetry::metrics::v1::{metric, number_data_point, *};
use session::protocol_ctx::{MetricType, OtlpMetricCtx};
use table::requests::{
METADATA_QUALITY_DECLARED, SEMANTIC_METRIC_METADATA_QUALITY, SEMANTIC_METRIC_ORIGINAL_NAME,
SEMANTIC_METRIC_TEMPORALITY, SEMANTIC_METRIC_TYPE, SEMANTIC_METRIC_UNIT,
METADATA_QUALITY_DECLARED, METRIC_TEMPORALITY_CUMULATIVE, METRIC_TEMPORALITY_DELTA,
SEMANTIC_METRIC_METADATA_QUALITY, SEMANTIC_METRIC_ORIGINAL_NAME, SEMANTIC_METRIC_TEMPORALITY,
SEMANTIC_METRIC_TYPE, SEMANTIC_METRIC_UNIT,
};
use crate::error::{self, Result};
@@ -285,8 +289,8 @@ fn temporality_value(data: &metric::Data) -> Option<&'static str> {
_ => return None,
};
match AggregationTemporality::try_from(raw) {
Ok(AggregationTemporality::Delta) => Some("delta"),
Ok(AggregationTemporality::Cumulative) => Some("cumulative"),
Ok(AggregationTemporality::Delta) => Some(METRIC_TEMPORALITY_DELTA),
Ok(AggregationTemporality::Cumulative) => Some(METRIC_TEMPORALITY_CUMULATIVE),
_ => None,
}
}
@@ -509,18 +513,15 @@ fn encode_metrics(
add_accepted_data_points(outcome, summary.data_points.len())?;
!summary.data_points.is_empty()
}
metric::Data::Histogram(hist) => {
encode_histogram(
table_writer,
&name,
hist,
resource_attrs,
scope_attrs,
metric_ctx,
)?;
add_accepted_data_points(outcome, hist.data_points.len())?;
!hist.data_points.is_empty()
}
metric::Data::Histogram(hist) => encode_histogram(
table_writer,
&name,
hist,
resource_attrs,
scope_attrs,
metric_ctx,
outcome,
)?,
metric::Data::ExponentialHistogram(hist) => encode_exponential_histogram(
table_writer,
&name,
@@ -932,9 +933,12 @@ fn write_attributes(
return Ok(());
};
let tags = attrs.iter().filter_map(|attr| {
let mut tags = Vec::with_capacity(attrs.len());
for attr in attrs {
// TODO(sunng87): allow different type of values
let value = scalar_value_string(attr.value.as_ref())?;
let Some(value) = scalar_value_string(attr.value.as_ref()) else {
continue;
};
let key = match attribute_type {
AttributeType::Resource | AttributeType::DataPoint => {
translate_label_name(&attr.key, metric_ctx.metric_translation_strategy)
@@ -947,9 +951,18 @@ fn write_attributes(
}
AttributeType::Legacy => legacy_normalize_otlp_name(&attr.key),
};
Some((key, value))
});
row_writer::write_tags(writer, tags, row)?;
if key == OTLP_AGGREGATION_TEMPORALITY_LABEL {
return Err(error::InvalidOtlpMetricInputSnafu {
reason: format!(
"OTLP attribute `{}` resolves to reserved label `{}`",
attr.key, OTLP_AGGREGATION_TEMPORALITY_LABEL
),
}
.build());
}
tags.push((key, value));
}
row_writer::write_tags(writer, tags.into_iter(), row)?;
Ok(())
}
@@ -998,6 +1011,26 @@ fn write_data_point_value(
Ok(())
}
fn write_temporality_tag(
table: &mut TableData,
row: &mut Vec<Value>,
is_delta: bool,
) -> Result<()> {
if is_delta {
row_writer::write_tag(
table,
OTLP_AGGREGATION_TEMPORALITY_LABEL,
GREPTIME_TEMPORALITY_DELTA,
row,
)?;
}
Ok(())
}
fn has_no_recorded_value(flags: u32) -> bool {
flags & DataPointFlags::NoRecordedValueMask as u32 != 0
}
fn write_tags_and_timestamp(
table: &mut TableData,
row: &mut Vec<Value>,
@@ -1086,7 +1119,6 @@ fn encode_gauge(
/// encode this sum metric
///
/// `aggregation_temporality` and `monotonic` are ignored for now
fn encode_sum(
table_writer: &mut MultiTableData,
name: &str,
@@ -1095,6 +1127,10 @@ fn encode_sum(
scope_attrs: Option<&Vec<KeyValue>>,
metric_ctx: &OtlpMetricCtx,
) -> Result<()> {
let is_delta = matches!(
AggregationTemporality::try_from(sum.aggregation_temporality),
Ok(AggregationTemporality::Delta)
);
let table = table_writer.get_or_default_table_data(
name,
APPROXIMATE_COLUMN_COUNT,
@@ -1112,7 +1148,17 @@ fn encode_sum(
data_point.time_unix_nano as i64,
metric_ctx,
)?;
write_data_point_value(table, &mut row, greptime_value(), &data_point.value)?;
write_temporality_tag(table, &mut row, is_delta)?;
if has_no_recorded_value(data_point.flags) {
row_writer::write_f64(
table,
greptime_value(),
f64::from_bits(PROMETHEUS_STALE_NAN_BITS),
&mut row,
)?;
} else {
write_data_point_value(table, &mut row, greptime_value(), &data_point.value)?;
}
table.add_row(row);
}
@@ -1139,22 +1185,71 @@ fn encode_histogram(
resource_attrs: Option<&Vec<KeyValue>>,
scope_attrs: Option<&Vec<KeyValue>>,
metric_ctx: &OtlpMetricCtx,
) -> Result<()> {
outcome: &mut MetricsIngestOutcome,
) -> Result<bool> {
let normalized_name = name;
let bucket_table_name = format!("{}{}", normalized_name, BUCKET_TABLE_SUFFIX);
let sum_table_name = format!("{}{}", normalized_name, SUM_TABLE_SUFFIX);
let count_table_name = format!("{}{}", normalized_name, COUNT_TABLE_SUFFIX);
let data_points_len = hist.data_points.len();
for data_point in &hist.data_points {
let bucket_table = table_writer.get_or_default_table_data(
&bucket_table_name,
APPROXIMATE_COLUMN_COUNT,
data_points_len * 3,
);
let mut accumulated_count = 0;
for (idx, count) in data_point.bucket_counts.iter().enumerate() {
let is_delta = matches!(
AggregationTemporality::try_from(hist.aggregation_temporality),
Ok(AggregationTemporality::Delta)
);
let stale_value = f64::from_bits(PROMETHEUS_STALE_NAN_BITS);
let mut emitted = false;
for (index, data_point) in hist.data_points.iter().enumerate() {
if let Some(reason) = histogram_data_point_rejection(data_point, is_delta) {
reject_data_points(outcome, 1, || {
format!("metric `{name}` data point {index}: {reason}")
})?;
continue;
}
let bucket_table =
table_writer.get_or_default_table_data(&bucket_table_name, APPROXIMATE_COLUMN_COUNT, 0);
let no_recorded_value = has_no_recorded_value(data_point.flags);
if no_recorded_value {
bucket_table.reserve_rows(data_point.explicit_bounds.len());
bucket_table.reserve_rows(1);
} else {
bucket_table.reserve_rows(data_point.bucket_counts.len().max(1));
}
let bucket_values = if no_recorded_value {
data_point
.explicit_bounds
.iter()
.copied()
.chain(std::iter::once(f64::INFINITY))
.map(|bound| (bound, stale_value))
.collect::<Vec<_>>()
} else if data_point.bucket_counts.is_empty() && data_point.explicit_bounds.is_empty() {
// OTLP count-only histograms map to one implicit Prometheus infinity bucket.
vec![(f64::INFINITY, data_point.count as f64)]
} else {
let mut accumulated_count = 0u64;
let mut values = Vec::with_capacity(data_point.bucket_counts.len());
for (idx, count) in data_point.bucket_counts.iter().enumerate() {
accumulated_count = accumulated_count.checked_add(*count).ok_or_else(|| {
error::InvalidParameterSnafu {
reason: format!(
"metric `{name}` data point {index}: bucket prefix overflows u64"
),
}
.build()
})?;
let bound =
data_point.explicit_bounds.get(idx).copied().or_else(|| {
(idx == data_point.explicit_bounds.len()).then_some(f64::INFINITY)
});
if let Some(bound) = bound {
values.push((bound, accumulated_count as f64));
}
}
values
};
for (bound, value) in bucket_values {
let mut bucket_row = bucket_table.alloc_one_row();
write_tags_and_timestamp(
bucket_table,
@@ -1165,31 +1260,9 @@ fn encode_histogram(
data_point.time_unix_nano as i64,
metric_ctx,
)?;
if let Some(upper_bounds) = data_point.explicit_bounds.get(idx) {
row_writer::write_tag(
bucket_table,
HISTOGRAM_LE_COLUMN,
upper_bounds,
&mut bucket_row,
)?;
} else if idx == data_point.explicit_bounds.len() {
// The last bucket
row_writer::write_tag(
bucket_table,
HISTOGRAM_LE_COLUMN,
f64::INFINITY,
&mut bucket_row,
)?;
}
accumulated_count += count;
row_writer::write_f64(
bucket_table,
greptime_value(),
accumulated_count as f64,
&mut bucket_row,
)?;
write_temporality_tag(bucket_table, &mut bucket_row, is_delta)?;
row_writer::write_tag(bucket_table, HISTOGRAM_LE_COLUMN, bound, &mut bucket_row)?;
row_writer::write_f64(bucket_table, greptime_value(), value, &mut bucket_row)?;
bucket_table.add_row(bucket_row);
}
@@ -1198,7 +1271,7 @@ fn encode_histogram(
let sum_table = table_writer.get_or_default_table_data(
&sum_table_name,
APPROXIMATE_COLUMN_COUNT,
data_points_len,
hist.data_points.len(),
);
let mut sum_row = sum_table.alloc_one_row();
write_tags_and_timestamp(
@@ -1210,15 +1283,20 @@ fn encode_histogram(
data_point.time_unix_nano as i64,
metric_ctx,
)?;
row_writer::write_f64(sum_table, greptime_value(), sum, &mut sum_row)?;
write_temporality_tag(sum_table, &mut sum_row, is_delta)?;
row_writer::write_f64(
sum_table,
greptime_value(),
if no_recorded_value { stale_value } else { sum },
&mut sum_row,
)?;
sum_table.add_row(sum_row);
}
let count_table = table_writer.get_or_default_table_data(
&count_table_name,
APPROXIMATE_COLUMN_COUNT,
data_points_len,
hist.data_points.len(),
);
let mut count_row = count_table.alloc_one_row();
write_tags_and_timestamp(
@@ -1230,17 +1308,75 @@ fn encode_histogram(
data_point.time_unix_nano as i64,
metric_ctx,
)?;
write_temporality_tag(count_table, &mut count_row, is_delta)?;
row_writer::write_f64(
count_table,
greptime_value(),
data_point.count as f64,
if no_recorded_value {
stale_value
} else {
data_point.count as f64
},
&mut count_row,
)?;
count_table.add_row(count_row);
add_accepted_data_points(outcome, 1)?;
emitted = true;
}
Ok(())
Ok(emitted)
}
pub(crate) fn histogram_data_point_rejection(
data_point: &HistogramDataPoint,
is_delta: bool,
) -> Option<String> {
if has_no_recorded_value(data_point.flags) {
return None;
}
if is_delta {
let valid_empty_layout =
data_point.bucket_counts.is_empty() && data_point.explicit_bounds.is_empty();
let expected_buckets = data_point.explicit_bounds.len().checked_add(1);
if !valid_empty_layout && expected_buckets != Some(data_point.bucket_counts.len()) {
return Some(format!(
"bucket_counts length {} must equal explicit_bounds length {} plus one",
data_point.bucket_counts.len(),
data_point.explicit_bounds.len()
));
}
if data_point
.explicit_bounds
.iter()
.any(|bound| !bound.is_finite())
|| data_point
.explicit_bounds
.windows(2)
.any(|bounds| bounds[0] >= bounds[1])
{
return Some("explicit_bounds must be finite and strictly increasing".to_string());
}
if data_point.count == 0 && data_point.sum.is_some_and(|sum| sum != 0.0) {
return Some("sum must be absent or zero when count is zero".to_string());
}
}
let bucket_total = data_point
.bucket_counts
.iter()
.try_fold(0u64, |total, count| total.checked_add(*count));
let Some(bucket_total) = bucket_total else {
return Some("bucket prefix overflows u64".to_string());
};
if is_delta && !data_point.bucket_counts.is_empty() && bucket_total != data_point.count {
return Some(format!(
"buckets contain {bucket_total} observations, declared count is {}",
data_point.count
));
}
None
}
fn encode_summary(
@@ -1388,6 +1524,8 @@ mod tests {
use super::*;
mod delta;
fn keyvalue(key: &str, value: &str) -> KeyValue {
KeyValue {
key: key.into(),
@@ -1786,6 +1924,7 @@ mod tests {
#[test]
fn test_encode_histogram() {
let mut tables = MultiTableData::default();
let mut outcome = MetricsIngestOutcome::default();
let data_points = vec![HistogramDataPoint {
attributes: vec![keyvalue("host", "testserver")],
@@ -1811,15 +1950,17 @@ mod tests {
Some(&vec![]),
Some(&vec![keyvalue("scope", "otel")]),
&OtlpMetricCtx::default(),
&mut outcome,
)
.unwrap();
assert_eq!(3, tables.num_tables());
assert_eq!(1, outcome.accepted_data_points);
// bucket table
let bucket_table = tables.get_or_default_table_data("histo_bucket", 0, 0);
assert_eq!(bucket_table.num_rows(), 5);
assert_eq!(bucket_table.num_columns(), 5);
assert_eq!(bucket_table.num_columns(), 6);
assert_eq!(
bucket_table
.columns()
@@ -1830,6 +1971,7 @@ mod tests {
"otel_scope_scope",
"host",
greptime_timestamp(),
OTLP_AGGREGATION_TEMPORALITY_LABEL,
"le",
greptime_value(),
]
@@ -1837,7 +1979,7 @@ mod tests {
let sum_table = tables.get_or_default_table_data("histo_sum", 0, 0);
assert_eq!(sum_table.num_rows(), 1);
assert_eq!(sum_table.num_columns(), 4);
assert_eq!(sum_table.num_columns(), 5);
assert_eq!(
sum_table
.columns()
@@ -1848,13 +1990,14 @@ mod tests {
"otel_scope_scope",
"host",
greptime_timestamp(),
OTLP_AGGREGATION_TEMPORALITY_LABEL,
greptime_value()
]
);
let count_table = tables.get_or_default_table_data("histo_count", 0, 0);
assert_eq!(count_table.num_rows(), 1);
assert_eq!(count_table.num_columns(), 4);
assert_eq!(count_table.num_columns(), 5);
assert_eq!(
count_table
.columns()
@@ -1865,6 +2008,7 @@ mod tests {
"otel_scope_scope",
"host",
greptime_timestamp(),
OTLP_AGGREGATION_TEMPORALITY_LABEL,
greptime_value()
]
);
src/servers/src/otlp/metrics/resource_info.rs
+20 -3
View File
@@ -27,13 +27,16 @@ use common_catalog::consts::SEMANTIC_GRAPH_WINDOW_NANOS;
use common_grpc::precision::Precision;
use common_query::prelude::{greptime_timestamp, greptime_value};
use otel_arrow_rust::proto::opentelemetry::common::v1::KeyValue;
use otel_arrow_rust::proto::opentelemetry::metrics::v1::{ResourceMetrics, metric};
use otel_arrow_rust::proto::opentelemetry::metrics::v1::{
AggregationTemporality, ResourceMetrics, metric,
};
use session::protocol_ctx::OtlpMetricCtx;
use crate::error::Result;
use crate::otlp::metrics::{
INSTANCE_KEY, JOB_KEY, ServiceIdentity, exponential_histogram_gate,
exponential_histogram_value, scalar_value_string, service_identity,
exponential_histogram_value, histogram_data_point_rejection, scalar_value_string,
service_identity,
};
use crate::otlp::trace::{
KEY_CONTAINER_ID, KEY_CONTAINER_NAME, KEY_HOST_ID, KEY_HOST_NAME, KEY_K8S_CONTAINER_NAME,
@@ -193,7 +196,19 @@ fn for_each_encoded_time(
visit_all(s.data_points.iter().map(|p| p.time_unix_nano), &mut visit)
}
Some(metric::Data::Histogram(h)) => {
visit_all(h.data_points.iter().map(|p| p.time_unix_nano), &mut visit)
let is_delta = matches!(
AggregationTemporality::try_from(h.aggregation_temporality),
Ok(AggregationTemporality::Delta)
);
visit_all(
h.data_points
.iter()
.filter(|point| {
histogram_data_point_rejection(point, is_delta).is_none()
})
.map(|point| point.time_unix_nano),
&mut visit,
)
}
Some(metric::Data::Summary(s)) => {
visit_all(s.data_points.iter().map(|p| p.time_unix_nano), &mut visit)
@@ -227,6 +242,8 @@ mod tests {
use super::*;
mod delta;
fn kv(key: &str, value: &str) -> KeyValue {
KeyValue {
key: key.into(),
src/servers/src/otlp/metrics/resource_info/tests/delta.rs
+70
View File
@@ -0,0 +1,70 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use otel_arrow_rust::proto::opentelemetry::metrics::v1::{
DataPointFlags, Histogram, HistogramDataPoint,
};
use super::*;
#[test]
fn observe_ignores_rejected_classic_histogram_windows() {
let window = SEMANTIC_GRAPH_WINDOW_NANOS;
let resource = ResourceMetrics {
scope_metrics: vec![ScopeMetrics {
metrics: vec![Metric {
data: Some(metric::Data::Histogram(Histogram {
data_points: vec![
HistogramDataPoint {
time_unix_nano: (window + 1) as u64,
count: 1,
bucket_counts: vec![1],
..Default::default()
},
HistogramDataPoint {
time_unix_nano: (3 * window + 1) as u64,
count: 1,
bucket_counts: vec![1],
explicit_bounds: vec![1.0, 2.0],
..Default::default()
},
HistogramDataPoint {
time_unix_nano: (4 * window + 1) as u64,
count: u64::MAX,
bucket_counts: vec![u64::MAX, 1],
flags: DataPointFlags::NoRecordedValueMask as u32,
..Default::default()
},
],
aggregation_temporality: AggregationTemporality::Delta as i32,
})),
..Default::default()
}],
..Default::default()
}],
..Default::default()
};
let mut data = ResourceInfoData::default();
data.observe(
&[kv("service.name", "api")],
&resource,
&OtlpMetricCtx::default(),
);
let windows = data.rows.values().next().unwrap();
assert_eq!(
vec![window + 1, 4 * window + 1],
windows.values().copied().collect::<Vec<_>>()
);
}
src/servers/src/otlp/metrics/tests/delta.rs
+394
View File
@@ -0,0 +1,394 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use otel_arrow_rust::proto::opentelemetry::common::v1::InstrumentationScope;
use super::*;
fn table_rows<'a>(request: &'a RowInsertRequests, table: &str) -> &'a api::v1::Rows {
request
.inserts
.iter()
.find(|insert| insert.table_name == table)
.unwrap_or_else(|| panic!("missing table {table}"))
.rows
.as_ref()
.unwrap()
}
fn column_index(rows: &api::v1::Rows, column: &str) -> usize {
rows.schema
.iter()
.position(|schema| schema.column_name == column)
.unwrap_or_else(|| panic!("missing column {column}"))
}
#[test]
fn test_raw_delta_sum_identity_and_stale_marker() {
set_default_prefix(Some("custom")).unwrap();
assert_eq!(
OTLP_AGGREGATION_TEMPORALITY_LABEL,
"otlp_aggregation_temporality"
);
let points = vec![
NumberDataPoint {
attributes: vec![keyvalue("host", "a")],
time_unix_nano: 1_000_000,
value: Some(Value::AsInt(10)),
..Default::default()
},
NumberDataPoint {
attributes: vec![keyvalue("host", "a")],
time_unix_nano: 2_000_000,
value: Some(Value::AsDouble(20.5)),
..Default::default()
},
NumberDataPoint {
attributes: vec![keyvalue("host", "a")],
time_unix_nano: 3_000_000,
value: Some(Value::AsDouble(99.0)),
flags: DataPointFlags::NoRecordedValueMask as u32,
..Default::default()
},
];
let metric = Metric {
name: "requests".to_string(),
data: Some(metric::Data::Sum(Sum {
data_points: points,
aggregation_temporality: AggregationTemporality::Delta as i32,
is_monotonic: true,
})),
..Default::default()
};
let conversion =
to_grpc_insert_requests(metrics_request(vec![metric]), &mut OtlpMetricCtx::default())
.unwrap();
assert_eq!(3, conversion.outcome.accepted_data_points);
assert_eq!(0, conversion.outcome.rejected_data_points);
let rows = table_rows(&conversion.requests, "requests_total");
let value = column_index(rows, greptime_value());
let temporality = column_index(rows, OTLP_AGGREGATION_TEMPORALITY_LABEL);
let host = column_index(rows, "host");
let values = rows
.rows
.iter()
.map(|row| row.values[value].value_data.as_ref().unwrap())
.collect::<Vec<_>>();
assert_eq!(Some(&ValueData::F64Value(10.0)), values.first().copied());
assert_eq!(Some(&ValueData::F64Value(20.5)), values.get(1).copied());
let ValueData::F64Value(stale) = values[2] else {
panic!("expected stale float")
};
assert_eq!(PROMETHEUS_STALE_NAN_BITS, stale.to_bits());
for row in &rows.rows {
assert_eq!(
Some(&ValueData::StringValue(
GREPTIME_TEMPORALITY_DELTA.to_string()
)),
row.values[temporality].value_data.as_ref()
);
assert_eq!(
Some(&ValueData::StringValue("a".to_string())),
row.values[host].value_data.as_ref()
);
}
for temporality in [
AggregationTemporality::Cumulative as i32,
AggregationTemporality::Unspecified as i32,
] {
let metric = Metric {
name: format!("sum_{temporality}"),
data: Some(metric::Data::Sum(Sum {
data_points: vec![NumberDataPoint::default()],
aggregation_temporality: temporality,
..Default::default()
})),
..Default::default()
};
let conversion =
to_grpc_insert_requests(metrics_request(vec![metric]), &mut OtlpMetricCtx::default())
.unwrap();
assert!(
conversion.requests.inserts[0]
.rows
.as_ref()
.unwrap()
.schema
.iter()
.all(|column| column.column_name != OTLP_AGGREGATION_TEMPORALITY_LABEL)
);
}
}
#[test]
fn test_delta_histogram_partial_rejection_and_inline_tombstone() {
let valid = HistogramDataPoint {
time_unix_nano: 1_000_000,
count: 10,
sum: Some(12.0),
bucket_counts: vec![2, 3, 5],
explicit_bounds: vec![1.0, 2.0],
..Default::default()
};
let malformed = HistogramDataPoint {
time_unix_nano: 2_000_000,
count: 10,
bucket_counts: vec![10],
explicit_bounds: vec![1.0, 2.0],
..Default::default()
};
let tombstone = HistogramDataPoint {
time_unix_nano: 3_000_000,
count: u64::MAX,
sum: Some(99.0),
bucket_counts: vec![u64::MAX, 1],
explicit_bounds: vec![3.0, 2.0],
flags: DataPointFlags::NoRecordedValueMask as u32,
..Default::default()
};
let metric = Metric {
name: "latency".to_string(),
data: Some(metric::Data::Histogram(Histogram {
data_points: vec![valid, malformed, tombstone],
aggregation_temporality: AggregationTemporality::Delta as i32,
})),
..Default::default()
};
let conversion =
to_grpc_insert_requests(metrics_request(vec![metric]), &mut OtlpMetricCtx::default())
.unwrap();
assert_eq!(2, conversion.outcome.accepted_data_points);
assert_eq!(1, conversion.outcome.rejected_data_points);
assert!(
conversion
.outcome
.error_message
.unwrap()
.contains("bucket_counts length")
);
let buckets = table_rows(&conversion.requests, "latency_bucket");
assert_eq!(6, buckets.rows.len());
let value = column_index(buckets, greptime_value());
let le = column_index(buckets, HISTOGRAM_LE_COLUMN);
let temporality = column_index(buckets, OTLP_AGGREGATION_TEMPORALITY_LABEL);
let ordinary = buckets.rows[..3]
.iter()
.map(|row| row.values[value].value_data.clone())
.collect::<Vec<_>>();
assert_eq!(
vec![
Some(ValueData::F64Value(2.0)),
Some(ValueData::F64Value(5.0)),
Some(ValueData::F64Value(10.0)),
],
ordinary
);
let tombstone_bounds = buckets.rows[3..]
.iter()
.map(|row| row.values[le].value_data.clone())
.collect::<Vec<_>>();
assert_eq!(
vec![
Some(ValueData::StringValue("3".to_string())),
Some(ValueData::StringValue("2".to_string())),
Some(ValueData::StringValue("inf".to_string())),
],
tombstone_bounds
);
for row in &buckets.rows {
assert_eq!(
Some(&ValueData::StringValue(
GREPTIME_TEMPORALITY_DELTA.to_string()
)),
row.values[temporality].value_data.as_ref()
);
}
for row in &buckets.rows[3..] {
let Some(ValueData::F64Value(value)) = row.values[value].value_data else {
panic!("expected stale float")
};
assert_eq!(PROMETHEUS_STALE_NAN_BITS, value.to_bits());
}
for table in ["latency_sum", "latency_count"] {
let rows = table_rows(&conversion.requests, table);
assert_eq!(2, rows.rows.len());
let value = column_index(rows, greptime_value());
let Some(ValueData::F64Value(stale)) = rows.rows[1].values[value].value_data else {
panic!("expected stale float")
};
assert_eq!(PROMETHEUS_STALE_NAN_BITS, stale.to_bits());
}
}
#[test]
fn test_histogram_validation_preserves_supported_siblings() {
for temporality in [
AggregationTemporality::Delta,
AggregationTemporality::Cumulative,
] {
let overflow = HistogramDataPoint {
count: u64::MAX,
bucket_counts: vec![u64::MAX, 1],
explicit_bounds: vec![1.0],
..Default::default()
};
let count_only = HistogramDataPoint {
count: 2,
sum: Some(3.0),
..Default::default()
};
let metric = Metric {
name: format!("hist_{temporality:?}"),
data: Some(metric::Data::Histogram(Histogram {
data_points: vec![overflow, count_only],
aggregation_temporality: temporality as i32,
})),
..Default::default()
};
let conversion =
to_grpc_insert_requests(metrics_request(vec![metric]), &mut OtlpMetricCtx::default())
.unwrap();
assert_eq!(1, conversion.outcome.accepted_data_points);
assert_eq!(1, conversion.outcome.rejected_data_points);
assert!(
conversion
.outcome
.error_message
.unwrap()
.contains("overflows u64")
);
let count = conversion
.requests
.inserts
.iter()
.find(|insert| insert.table_name.ends_with(COUNT_TABLE_SUFFIX))
.unwrap()
.rows
.as_ref()
.unwrap();
assert_eq!(1, count.rows.len());
let buckets = conversion
.requests
.inserts
.iter()
.find(|insert| insert.table_name.ends_with(BUCKET_TABLE_SUFFIX))
.unwrap()
.rows
.as_ref()
.unwrap();
assert_eq!(1, buckets.rows.len());
let le = column_index(buckets, HISTOGRAM_LE_COLUMN);
let value = column_index(buckets, greptime_value());
assert_eq!(
Some(&ValueData::StringValue("inf".to_string())),
buckets.rows[0].values[le].value_data.as_ref()
);
assert_eq!(
Some(&ValueData::F64Value(2.0)),
buckets.rows[0].values[value].value_data.as_ref()
);
}
}
#[test]
fn test_reserved_temporality_label_uses_final_persisted_key() {
set_default_prefix(Some("custom")).unwrap();
let gauge = |attributes: Vec<KeyValue>| Metric {
name: "gauge".to_string(),
data: Some(metric::Data::Gauge(Gauge {
data_points: vec![NumberDataPoint {
attributes,
..Default::default()
}],
})),
..Default::default()
};
let error = to_grpc_insert_requests(
metrics_request(vec![gauge(vec![keyvalue(
OTLP_AGGREGATION_TEMPORALITY_LABEL,
"user",
)])]),
&mut OtlpMetricCtx::default(),
)
.unwrap_err();
assert!(matches!(error, error::Error::InvalidOtlpMetricInput { .. }));
let mut request = metrics_request(vec![gauge(vec![])]);
request.resource_metrics[0].scope_metrics[0].scope = Some(InstrumentationScope {
attributes: vec![keyvalue(OTLP_AGGREGATION_TEMPORALITY_LABEL, "safe")],
..Default::default()
});
let mut ctx = OtlpMetricCtx {
promote_scope_attrs: true,
..Default::default()
};
let conversion = to_grpc_insert_requests(request, &mut ctx).unwrap();
assert!(
column_names(&conversion.requests, "gauge").contains(&format!(
"otel_scope_{}",
OTLP_AGGREGATION_TEMPORALITY_LABEL
))
);
}
#[test]
fn test_histogram_semantics_follow_only_emitted_rows() {
let cumulative = histogram_metric("latency");
let mut delta = histogram_metric("latency");
let Some(metric::Data::Histogram(histogram)) = delta.data.as_mut() else {
unreachable!()
};
histogram.aggregation_temporality = AggregationTemporality::Delta as i32;
let conversion = to_grpc_insert_requests(
metrics_request(vec![cumulative.clone(), delta.clone()]),
&mut OtlpMetricCtx::default(),
)
.unwrap();
let semantics = decode(&conversion.semantic_index);
for table in ["latency_bucket", "latency_sum", "latency_count"] {
assert_eq!(
Some("mixed"),
semantics[table]
.get(SEMANTIC_METRIC_TEMPORALITY)
.map(String::as_str)
);
}
let Some(metric::Data::Histogram(histogram)) = delta.data.as_mut() else {
unreachable!()
};
histogram.data_points[0].explicit_bounds = vec![1.0];
let conversion = to_grpc_insert_requests(
metrics_request(vec![cumulative, delta]),
&mut OtlpMetricCtx::default(),
)
.unwrap();
assert_eq!(1, conversion.outcome.rejected_data_points);
let semantics = decode(&conversion.semantic_index);
for table in ["latency_bucket", "latency_sum", "latency_count"] {
assert_eq!(
Some(METRIC_TEMPORALITY_CUMULATIVE),
semantics[table]
.get(SEMANTIC_METRIC_TEMPORALITY)
.map(String::as_str)
);
}
}
src/table/src/requests/semantic.rs
+9 -3
View File
@@ -68,8 +68,8 @@ pub const SEMANTIC_METRIC_TYPE: &str = "greptime.semantic.metric.type";
/// UCUM unit, e.g. `s`, `By`, `{request}`. Discarded by the row encoders, so it
/// is unrecoverable once ingested.
pub const SEMANTIC_METRIC_UNIT: &str = "greptime.semantic.metric.unit";
/// `cumulative` / `delta` (OTel only). Invisible in the metric name, so it is
/// unrecoverable from the table alone.
/// Catalog-level `cumulative` / `delta` / `mixed` description for OTLP metrics.
/// Per-series query behavior is determined from stored row identity instead.
pub const SEMANTIC_METRIC_TEMPORALITY: &str = "greptime.semantic.metric.temporality";
/// [`METADATA_QUALITY_DECLARED`] when the protocol stated the type, or
/// [`METADATA_QUALITY_INFERRED`] when guessed from a name suffix.
@@ -135,6 +135,9 @@ pub const SOURCE_ELASTICSEARCH: &str = "elasticsearch";
pub const METADATA_QUALITY_DECLARED: &str = "declared";
pub const METADATA_QUALITY_INFERRED: &str = "inferred";
pub const METRIC_TEMPORALITY_CUMULATIVE: &str = "cumulative";
pub const METRIC_TEMPORALITY_DELTA: &str = "delta";
/// Sentinel for a key that cannot be determined at stamp time.
pub const SEMANTIC_VALUE_UNKNOWN: &str = "unknown";
/// Sentinel for a single-valued key that saw conflicting sources.
@@ -277,7 +280,10 @@ pub fn validate_semantic_option(key: &str, value: &str) -> bool {
| "unknown"
),
SEMANTIC_METRIC_TEMPORALITY => {
matches!(value, "cumulative" | "delta" | "mixed" | "unknown")
matches!(
value,
METRIC_TEMPORALITY_CUMULATIVE | METRIC_TEMPORALITY_DELTA | "mixed" | "unknown"
)
}
SEMANTIC_METRIC_METADATA_QUALITY => matches!(value, "declared" | "inferred" | "unknown"),
tests-integration/src/otlp.rs
+368 -2
View File
@@ -26,8 +26,8 @@ mod test {
};
use otel_arrow_rust::proto::opentelemetry::metrics::v1::number_data_point::Value;
use otel_arrow_rust::proto::opentelemetry::metrics::v1::{
Gauge, Histogram, HistogramDataPoint, Metric, NumberDataPoint, ResourceMetrics,
ScopeMetrics, metric,
AggregationTemporality, DataPointFlags, Gauge, Histogram, HistogramDataPoint, Metric,
NumberDataPoint, ResourceMetrics, ScopeMetrics, Sum, metric,
};
use otel_arrow_rust::proto::opentelemetry::resource::v1::Resource;
use servers::query_handler::OpenTelemetryProtocolHandler;
@@ -54,6 +54,51 @@ mod test {
test_otlp(&instance.frontend()).await;
}
#[tokio::test(flavor = "multi_thread")]
pub async fn test_otlp_fixed_schema_rejects_missing_temporality_tag() {
let standalone = GreptimeDbStandaloneBuilder::new("test_otlp_fixed_schema")
.with_auto_create_table(false)
.build()
.await;
let instance = standalone.fe_instance();
let ctx = Arc::new(QueryContext::with(DEFAULT_CATALOG_NAME, "public"));
let mut output = instance
.do_query(
"CREATE TABLE fixed_delta_total (\
\"stream\" STRING, greptime_timestamp TIMESTAMP(3) NOT NULL, \
greptime_value DOUBLE, TIME INDEX (greptime_timestamp), \
PRIMARY KEY (\"stream\")) ENGINE=mito",
ctx.clone(),
)
.await;
let result = output.remove(0);
assert!(result.is_ok(), "{result:?}");
let error = instance
.metrics(
build_sum_request("fixed.delta", AggregationTemporality::Delta, &[(60, 10)]),
ctx.clone(),
)
.await
.unwrap_err();
assert!(format!("{error:?}").contains("otlp_aggregation_temporality"));
let mut output = instance
.do_query("SELECT COUNT(*) FROM fixed_delta_total", ctx)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
assert!(
RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap()
.contains("| 0 |")
);
}
async fn test_otlp(instance: &Arc<Instance>) {
let req = build_request();
let db = "otlp";
@@ -74,6 +119,272 @@ mod test {
let resp = instance.metrics(req, ctx.clone()).await;
assert!(resp.is_ok());
let mut output = instance
.do_query(
"CREATE TABLE raw_delta_mito_total (\
\"stream\" STRING, greptime_timestamp TIMESTAMP(3) NOT NULL, \
greptime_value DOUBLE, TIME INDEX (greptime_timestamp), \
PRIMARY KEY (\"stream\")) ENGINE=mito",
ctx.clone(),
)
.await;
let result = output.remove(0);
assert!(result.is_ok(), "{result:?}");
for (metric, table) in [
("raw.delta", "raw_delta_total"),
("raw.delta.mito", "raw_delta_mito_total"),
] {
for request in [
build_sum_request(metric, AggregationTemporality::Cumulative, &[(60, 10)]),
build_sum_request(
metric,
AggregationTemporality::Delta,
&[(60, 10), (120, 20), (180, 15)],
),
build_sum_request(metric, AggregationTemporality::Cumulative, &[(180, 30)]),
] {
let result = instance.metrics(request, ctx.clone()).await;
assert!(result.is_ok(), "{metric}: {result:?}");
}
let mut output = instance
.do_query(
&format!(
"SELECT COALESCE(otlp_aggregation_temporality, '') AS temporality, \
COUNT(*) AS samples, SUM(greptime_value) AS total \
FROM {table} GROUP BY otlp_aggregation_temporality ORDER BY temporality"
),
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
assert_eq!(
RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap(),
"\
+-------------+---------+-------+
| temporality | samples | total |
+-------------+---------+-------+
| | 2 | 40.0 |
| delta | 3 | 45.0 |
+-------------+---------+-------+"
);
for (function, expected) in [("increase", "45.0"), ("rate", "0.25")] {
let mut output = instance
.do_query(
&format!("TQL EVAL (180, 180, '1m') {function}({table}[3m])"),
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
let rendered = RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap();
assert!(rendered.contains("delta"), "{rendered}");
assert!(rendered.contains(expected), "{rendered}");
}
let mut stale = build_sum_request(metric, AggregationTemporality::Delta, &[(240, 99)]);
let Some(metric::Data::Sum(sum)) = stale.resource_metrics[0].scope_metrics[0].metrics
[0]
.data
.as_mut() else {
unreachable!()
};
sum.data_points[0].flags = DataPointFlags::NoRecordedValueMask as u32;
assert!(instance.metrics(stale, ctx.clone()).await.is_ok());
for (matcher, expected_rows) in [
("otlp_aggregation_temporality=\"delta\"", 0),
("otlp_aggregation_temporality!=\"delta\"", 1),
] {
let mut output = instance
.do_query(
&format!("TQL EVAL (240, 240, '1m') {table}{{{matcher}}}"),
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
let batches = RecordBatches::try_collect(stream).await.unwrap();
assert_eq!(
expected_rows,
batches.iter().map(|batch| batch.num_rows()).sum::<usize>()
);
if expected_rows == 1 {
assert!(batches.pretty_print().unwrap().contains("30.0"));
}
}
}
let malformed = ExportMetricsServiceRequest {
resource_metrics: vec![ResourceMetrics {
scope_metrics: vec![ScopeMetrics {
metrics: vec![Metric {
name: "rejected.delta.histogram".to_string(),
data: Some(metric::Data::Histogram(Histogram {
data_points: vec![HistogramDataPoint {
count: 1,
bucket_counts: vec![1],
explicit_bounds: vec![1.0, 2.0],
..Default::default()
}],
aggregation_temporality: AggregationTemporality::Delta as i32,
})),
..Default::default()
}],
..Default::default()
}],
..Default::default()
}],
};
let outcome = instance.metrics(malformed, ctx.clone()).await.unwrap();
assert_eq!(0, outcome.accepted_data_points);
assert_eq!(1, outcome.rejected_data_points);
assert!(
outcome
.error_message
.as_deref()
.unwrap()
.contains("bucket_counts length")
);
let mut output = instance
.do_query(
"SELECT COUNT(*) FROM information_schema.tables WHERE table_name IN \
('rejected_delta_histogram_bucket', 'rejected_delta_histogram_sum', \
'rejected_delta_histogram_count')",
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
assert!(
RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap()
.contains("| 0 |")
);
let point = |stream: &str, seconds: u64, bounds: Vec<f64>, sum| HistogramDataPoint {
attributes: vec![keyvalue("stream", stream)],
time_unix_nano: seconds * 1_000_000_000,
count: u64::try_from(bounds.len() + 1).unwrap(),
sum,
bucket_counts: vec![1; bounds.len() + 1],
explicit_bounds: bounds,
..Default::default()
};
let tombstone = |stream: &str, seconds: u64, bounds: Vec<f64>, sum| HistogramDataPoint {
flags: DataPointFlags::NoRecordedValueMask as u32,
..point(stream, seconds, bounds, sum)
};
for points in [
vec![
point("same", 300, vec![1.0, 2.0], Some(4.0)),
tombstone("same", 360, vec![1.0, 2.0], None),
],
vec![
point("changed", 420, vec![3.0, 5.0], Some(8.0)),
tombstone("changed", 480, vec![1.0, 2.0], Some(99.0)),
],
vec![
point("boundless", 420, vec![3.0, 5.0], Some(8.0)),
tombstone("boundless", 480, vec![], None),
],
vec![tombstone("new", 480, vec![], None)],
] {
let outcome = instance
.metrics(
build_histogram_request("raw.delta.histogram", points),
ctx.clone(),
)
.await
.unwrap();
assert_eq!(0, outcome.rejected_data_points);
}
let mut output = instance
.do_query(
"TQL EVAL (300, 300, '1m') \
histogram_quantile(0.5, rate(raw_delta_histogram_bucket[2m]))",
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
let rendered = RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap();
assert!(rendered.contains("1.5"), "{rendered}");
for (query, expected_rows) in [
("raw_delta_histogram_bucket{stream=\"same\",le=~\"1|2\"}", 0),
("raw_delta_histogram_sum{stream=\"same\"}", 1),
(
"raw_delta_histogram_bucket{stream=\"changed\",le=~\"3|5\"}",
2,
),
("raw_delta_histogram_sum{stream=\"changed\"}", 0),
(
"raw_delta_histogram_bucket{stream=\"boundless\",le=~\"3|5\"}",
2,
),
] {
let mut output = instance
.do_query(&format!("TQL EVAL (480, 480, '1m') {query}"), ctx.clone())
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
let batches = RecordBatches::try_collect(stream).await.unwrap();
assert_eq!(
expected_rows,
batches.iter().map(|batch| batch.num_rows()).sum::<usize>(),
"{query}: {}",
batches.pretty_print().unwrap()
);
}
let mut output = instance
.do_query(
"SELECT \
(SELECT COUNT(*) FROM raw_delta_histogram_bucket WHERE \"stream\" = 'new') AS buckets, \
(SELECT COUNT(*) FROM raw_delta_histogram_count WHERE \"stream\" = 'new') AS counts, \
(SELECT COUNT(*) FROM raw_delta_histogram_sum WHERE \"stream\" = 'new') AS sums",
ctx.clone(),
)
.await;
let OutputData::Stream(stream) = output.remove(0).unwrap().data else {
unreachable!()
};
let rendered = RecordBatches::try_collect(stream)
.await
.unwrap()
.pretty_print()
.unwrap();
assert!(
rendered.contains("| 1 | 1 | 0 |"),
"{rendered}"
);
let mut output = instance
.do_query(
"SELECT * FROM my_test_metric_my_ignored_unit ORDER BY greptime_timestamp",
@@ -232,6 +543,61 @@ mod test {
}
}
fn build_sum_request(
name: &str,
temporality: AggregationTemporality,
points: &[(u64, i64)],
) -> ExportMetricsServiceRequest {
let data_points = points
.iter()
.map(|(seconds, value)| NumberDataPoint {
attributes: vec![keyvalue("stream", "same")],
time_unix_nano: *seconds * 1_000_000_000,
value: Some(Value::AsInt(*value)),
..Default::default()
})
.collect();
ExportMetricsServiceRequest {
resource_metrics: vec![ResourceMetrics {
scope_metrics: vec![ScopeMetrics {
metrics: vec![Metric {
name: name.to_string(),
data: Some(metric::Data::Sum(Sum {
data_points,
aggregation_temporality: temporality as i32,
is_monotonic: true,
})),
..Default::default()
}],
..Default::default()
}],
..Default::default()
}],
}
}
fn build_histogram_request(
name: &str,
points: Vec<HistogramDataPoint>,
) -> ExportMetricsServiceRequest {
ExportMetricsServiceRequest {
resource_metrics: vec![ResourceMetrics {
scope_metrics: vec![ScopeMetrics {
metrics: vec![Metric {
name: name.to_string(),
data: Some(metric::Data::Histogram(Histogram {
data_points: points,
aggregation_temporality: AggregationTemporality::Delta as i32,
})),
..Default::default()
}],
..Default::default()
}],
..Default::default()
}],
}
}
fn keyvalue(key: &str, value: &str) -> KeyValue {
KeyValue {
key: key.into(),
tests-integration/tests/grpc.rs
+266 -2
View File
@@ -38,8 +38,9 @@ use common_runtime::Runtime;
use common_runtime::runtime::{BuilderBuild, RuntimeTrait};
use common_test_util::find_workspace_path;
use datatypes::arrow::array::{
Array, ArrayRef, Float64Array, Int32Array, ListBuilder, StringArray, StructArray,
TimestampNanosecondArray, UInt8Array, UInt16Array, UInt32Array, UInt64Array, UInt64Builder,
Array, ArrayRef, Float64Array, Float64Builder, Int32Array, ListBuilder, StringArray,
StructArray, TimestampNanosecondArray, UInt8Array, UInt16Array, UInt32Array, UInt64Array,
UInt64Builder,
};
use datatypes::arrow::datatypes::{DataType, Field};
use datatypes::arrow::ipc::writer::StreamWriter;
@@ -104,6 +105,7 @@ macro_rules! grpc_tests {
test_private_system_tables_auto_create_table_with_global_disabled,
test_private_system_tables_bypass_auto_create_hint,
test_otel_arrow_auth,
test_otel_arrow_delta_histogram,
test_otel_arrow_exponential_histogram,
test_insert_and_select,
test_dbname,
@@ -563,6 +565,268 @@ fn exponential_histogram_arrow_batch(batch_id: i64, scales: &[i32]) -> BatchArro
}
}
fn delta_histogram_arrow_batch(
batch_id: i64,
points: &[(u64, &[u64], &[f64])],
) -> BatchArrowRecords {
let resource = StructArray::from(vec![(
Arc::new(Field::new(arrow_consts::ID, DataType::UInt16, true)),
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
)]);
let scope = StructArray::from(vec![(
Arc::new(Field::new(arrow_consts::ID, DataType::UInt16, true)),
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
)]);
let metrics = ArrowRecordBatch::try_from_iter(vec![
(
arrow_consts::ID,
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
),
(arrow_consts::RESOURCE, Arc::new(resource) as ArrayRef),
(arrow_consts::SCOPE, Arc::new(scope) as ArrayRef),
(
arrow_consts::METRIC_TYPE,
Arc::new(UInt8Array::from(vec![ArrowMetricType::Histogram as u8])) as ArrayRef,
),
(
arrow_consts::NAME,
Arc::new(StringArray::from(vec!["otel.arrow.delta.histogram"])) as ArrayRef,
),
(
arrow_consts::AGGREGATION_TEMPORALITY,
Arc::new(Int32Array::from(vec![AggregationTemporality::Delta as i32])) as ArrayRef,
),
])
.unwrap();
let mut bucket_counts = ListBuilder::new(UInt64Builder::new());
let mut explicit_bounds = ListBuilder::new(Float64Builder::new());
for (_, counts, bounds) in points {
bucket_counts.values().append_slice(counts);
bucket_counts.append(true);
explicit_bounds.values().append_slice(bounds);
explicit_bounds.append(true);
}
let bucket_counts = bucket_counts.finish();
let explicit_bounds = explicit_bounds.finish();
let data_points = ArrowRecordBatch::try_from_iter(vec![
(
arrow_consts::ID,
Arc::new(UInt32Array::from_iter_values(
(0..points.len()).map(|id| u32::try_from(id).unwrap()),
)) as ArrayRef,
),
(
arrow_consts::PARENT_ID,
Arc::new(UInt16Array::from(vec![0_u16; points.len()])) as ArrayRef,
),
(
arrow_consts::START_TIME_UNIX_NANO,
Arc::new(TimestampNanosecondArray::from(vec![
1_000_000_000;
points.len()
])) as ArrayRef,
),
(
arrow_consts::TIME_UNIX_NANO,
Arc::new(TimestampNanosecondArray::from_iter_values(
(1..=points.len()).map(|second| i64::try_from(second).unwrap() * 1_000_000_000),
)) as ArrayRef,
),
(
arrow_consts::HISTOGRAM_COUNT,
Arc::new(UInt64Array::from(
points
.iter()
.map(|(count, _, _)| *count)
.collect::<Vec<_>>(),
)) as ArrayRef,
),
(
arrow_consts::HISTOGRAM_SUM,
Arc::new(Float64Array::from(vec![1.0; points.len()])) as ArrayRef,
),
(
arrow_consts::HISTOGRAM_BUCKET_COUNTS,
Arc::new(bucket_counts) as ArrayRef,
),
(
arrow_consts::HISTOGRAM_EXPLICIT_BOUNDS,
Arc::new(explicit_bounds) as ArrayRef,
),
(
arrow_consts::FLAGS,
Arc::new(UInt32Array::from(vec![0_u32; points.len()])) as ArrayRef,
),
])
.unwrap();
BatchArrowRecords {
batch_id,
arrow_payloads: vec![
ArrowPayload {
schema_id: format!("metrics-{batch_id}"),
r#type: ArrowPayloadType::UnivariateMetrics as i32,
record: serialize_arrow_record_batch(&metrics),
},
ArrowPayload {
schema_id: format!("histogram-{batch_id}"),
r#type: ArrowPayloadType::HistogramDataPoints as i32,
record: serialize_arrow_record_batch(&data_points),
},
],
headers: vec![],
}
}
fn gauge_arrow_batch(batch_id: i64, reserved_attr: bool) -> BatchArrowRecords {
let resource = StructArray::from(vec![(
Arc::new(Field::new(arrow_consts::ID, DataType::UInt16, true)),
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
)]);
let scope = StructArray::from(vec![(
Arc::new(Field::new(arrow_consts::ID, DataType::UInt16, true)),
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
)]);
let metrics = ArrowRecordBatch::try_from_iter(vec![
(
arrow_consts::ID,
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
),
(arrow_consts::RESOURCE, Arc::new(resource) as ArrayRef),
(arrow_consts::SCOPE, Arc::new(scope) as ArrayRef),
(
arrow_consts::METRIC_TYPE,
Arc::new(UInt8Array::from(vec![ArrowMetricType::Gauge as u8])) as ArrayRef,
),
(
arrow_consts::NAME,
Arc::new(StringArray::from(vec!["otel.arrow.gauge"])) as ArrayRef,
),
])
.unwrap();
let data_points = ArrowRecordBatch::try_from_iter(vec![
(
arrow_consts::ID,
Arc::new(UInt32Array::from(vec![0_u32])) as ArrayRef,
),
(
arrow_consts::PARENT_ID,
Arc::new(UInt16Array::from(vec![0_u16])) as ArrayRef,
),
(
arrow_consts::START_TIME_UNIX_NANO,
Arc::new(TimestampNanosecondArray::from(vec![1_000_000_000])) as ArrayRef,
),
(
arrow_consts::TIME_UNIX_NANO,
Arc::new(TimestampNanosecondArray::from(vec![2_000_000_000])) as ArrayRef,
),
(
arrow_consts::DOUBLE_VALUE,
Arc::new(Float64Array::from(vec![1.0])) as ArrayRef,
),
(
arrow_consts::FLAGS,
Arc::new(UInt32Array::from(vec![0_u32])) as ArrayRef,
),
])
.unwrap();
let mut arrow_payloads = vec![
ArrowPayload {
schema_id: format!("metrics-{batch_id}"),
r#type: ArrowPayloadType::UnivariateMetrics as i32,
record: serialize_arrow_record_batch(&metrics),
},
ArrowPayload {
schema_id: format!("number-{batch_id}"),
r#type: ArrowPayloadType::NumberDataPoints as i32,
record: serialize_arrow_record_batch(&data_points),
},
];
if reserved_attr {
let attributes = ArrowRecordBatch::try_from_iter(vec![
(
arrow_consts::PARENT_ID,
Arc::new(UInt32Array::from(vec![0_u32])) as ArrayRef,
),
(
arrow_consts::ATTRIBUTE_KEY,
Arc::new(StringArray::from(vec!["otlp_aggregation_temporality"])) as ArrayRef,
),
(
arrow_consts::ATTRIBUTE_TYPE,
Arc::new(UInt8Array::from(vec![1_u8])) as ArrayRef,
),
(
arrow_consts::ATTRIBUTE_STR,
Arc::new(StringArray::from(vec!["user"])) as ArrayRef,
),
])
.unwrap();
arrow_payloads.push(ArrowPayload {
schema_id: format!("number-attrs-{batch_id}"),
r#type: ArrowPayloadType::NumberDpAttrs as i32,
record: serialize_arrow_record_batch(&attributes),
});
}
BatchArrowRecords {
batch_id,
arrow_payloads,
headers: vec![],
}
}
pub async fn test_otel_arrow_delta_histogram(store_type: StorageType) {
let (_instance, server) =
setup_grpc_server(store_type, "test_otel_arrow_delta_histogram").await;
let addr = server.bind_addr().unwrap().to_string();
let mut client = ArrowMetricsServiceClient::connect(format!("http://{addr}"))
.await
.unwrap();
let valid = (3, &[1, 2][..], &[1.0][..]);
let malformed = (1, &[1][..], &[1.0][..]);
let request = Request::new(futures::stream::iter([
delta_histogram_arrow_batch(10, &[valid, malformed]),
delta_histogram_arrow_batch(11, &[malformed]),
delta_histogram_arrow_batch(12, &[valid]),
gauge_arrow_batch(13, true),
gauge_arrow_batch(14, false),
]));
let mut response = client.arrow_metrics(request).await.unwrap().into_inner();
let mixed = response.message().await.unwrap().unwrap();
assert_eq!(10, mixed.batch_id);
assert_eq!(ArrowStatusCode::Ok as i32, mixed.status_code);
assert!(mixed.status_message.contains("bucket_counts length"));
let rejected = response.message().await.unwrap().unwrap();
assert_eq!(11, rejected.batch_id);
assert_eq!(
ArrowStatusCode::InvalidArgument as i32,
rejected.status_code
);
assert!(rejected.status_message.contains("bucket_counts length"));
let later_valid = response.message().await.unwrap().unwrap();
assert_eq!(12, later_valid.batch_id);
assert_eq!(ArrowStatusCode::Ok as i32, later_valid.status_code);
assert!(later_valid.status_message.is_empty());
let collision = response.message().await.unwrap().unwrap();
assert_eq!(13, collision.batch_id);
assert_eq!(
ArrowStatusCode::InvalidArgument as i32,
collision.status_code
);
assert!(collision.status_message.contains("reserved label"));
let after_collision = response.message().await.unwrap().unwrap();
assert_eq!(14, after_collision.batch_id);
assert_eq!(ArrowStatusCode::Ok as i32, after_collision.status_code);
assert!(after_collision.status_message.is_empty());
let _ = server.shutdown().await;
}
pub async fn test_otel_arrow_exponential_histogram(store_type: StorageType) {
let (_instance, server) =
setup_grpc_server(store_type, "test_otel_arrow_exponential_histogram").await;
tests-integration/tests/http.rs
+114 -12
View File
@@ -6596,13 +6596,14 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// "otel_scope_name" STRING NULL,
// "otel_scope_schema_url" STRING NULL,
// "otel_scope_version" STRING NULL,
// "otlp_aggregation_temporality" STRING NULL,
// "service_name" STRING NULL,
// "service_version" STRING NULL,
// "session_id" STRING NULL,
// "terminal_type" STRING NULL,
// "user_id" STRING NULL,
// TIME INDEX ("greptime_timestamp"),
// PRIMARY KEY ("host_arch", "job", "model", "os_version", "otel_scope_name", "otel_scope_schema_url", "otel_scope_version", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// PRIMARY KEY ("host_arch", "job", "model", "os_version", "otel_scope_name", "otel_scope_schema_url", "otel_scope_version", "otlp_aggregation_temporality", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// )
// ENGINE=metric
// WITH(
@@ -6610,7 +6611,7 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// on_physical_table = 'greptime_physical_table',
// otlp_metric_compat = 'prom'
// )
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"host_arch\\\" STRING NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"os_version\\\" STRING NULL,\\n \\\"otel_scope_name\\\" STRING NULL,\\n \\\"otel_scope_schema_url\\\" STRING NULL,\\n \\\"otel_scope_version\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"host_arch\\\", \\\"job\\\", \\\"model\\\", \\\"os_version\\\", \\\"otel_scope_name\\\", \\\"otel_scope_schema_url\\\", \\\"otel_scope_version\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"host_arch\\\" STRING NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"os_version\\\" STRING NULL,\\n \\\"otel_scope_name\\\" STRING NULL,\\n \\\"otel_scope_schema_url\\\" STRING NULL,\\n \\\"otel_scope_version\\\" STRING NULL,\\n \\\"otlp_aggregation_temporality\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"host_arch\\\", \\\"job\\\", \\\"model\\\", \\\"os_version\\\", \\\"otel_scope_name\\\", \\\"otel_scope_schema_url\\\", \\\"otel_scope_version\\\", \\\"otlp_aggregation_temporality\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
validate_data(
"otlp_metrics_all_show_create_table",
&client,
@@ -6620,11 +6621,11 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
.await;
// select metrics data
let expected = "[[1753780559836,2.244618,\"arm64\",\"claude-code\",\"claude-sonnet-4-20250514\",\"25.0.0\",\"com.anthropic.claude_code\",\"\",\"1.0.62\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,0.0052544,\"arm64\",\"claude-code\",\"claude-3-5-haiku-20241022\",\"25.0.0\",\"com.anthropic.claude_code\",\"\",\"1.0.62\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
let expected = "[[1753780559836,2.244618,\"arm64\",\"claude-code\",\"claude-sonnet-4-20250514\",\"25.0.0\",\"com.anthropic.claude_code\",\"\",\"1.0.62\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,0.0052544,\"arm64\",\"claude-code\",\"claude-3-5-haiku-20241022\",\"25.0.0\",\"com.anthropic.claude_code\",\"\",\"1.0.62\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
validate_data(
"otlp_metrics_all_select",
&client,
"select * from `claude_code_cost_usage_USD_total`;",
"select * from `claude_code_cost_usage_USD_total` order by model desc;",
expected,
)
.await;
@@ -6701,13 +6702,14 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// "model" STRING NULL,
// "os_type" STRING NULL,
// "os_version" STRING NULL,
// "otlp_aggregation_temporality" STRING NULL,
// "service_name" STRING NULL,
// "service_version" STRING NULL,
// "session_id" STRING NULL,
// "terminal_type" STRING NULL,
// "user_id" STRING NULL,
// TIME INDEX ("greptime_timestamp"),
// PRIMARY KEY ("job", "model", "os_type", "os_version", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// PRIMARY KEY ("job", "model", "os_type", "os_version", "otlp_aggregation_temporality", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// )
// ENGINE=metric
// WITH(
@@ -6715,7 +6717,7 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// on_physical_table = 'greptime_physical_table',
// otlp_metric_compat = 'prom'
// )
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"os_type\\\" STRING NULL,\\n \\\"os_version\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"job\\\", \\\"model\\\", \\\"os_type\\\", \\\"os_version\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"os_type\\\" STRING NULL,\\n \\\"os_version\\\" STRING NULL,\\n \\\"otlp_aggregation_temporality\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"job\\\", \\\"model\\\", \\\"os_type\\\", \\\"os_version\\\", \\\"otlp_aggregation_temporality\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
validate_data(
"otlp_metrics_show_create_table",
&client,
@@ -6725,11 +6727,11 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
.await;
// select metrics data
let expected = "[[1753780559836,0.0052544,\"claude-code\",\"claude-3-5-haiku-20241022\",\"darwin\",\"25.0.0\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,2.244618,\"claude-code\",\"claude-sonnet-4-20250514\",\"darwin\",\"25.0.0\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
let expected = "[[1753780559836,2.244618,\"claude-code\",\"claude-sonnet-4-20250514\",\"darwin\",\"25.0.0\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,0.0052544,\"claude-code\",\"claude-3-5-haiku-20241022\",\"darwin\",\"25.0.0\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
validate_data(
"otlp_metrics_select",
&client,
"select * from `claude_code_cost_usage_USD_total`;",
"select * from `claude_code_cost_usage_USD_total` order by model desc;",
expected,
)
.await;
@@ -6765,13 +6767,14 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// "greptime_value" DOUBLE NULL,
// "job" STRING NULL,
// "model" STRING NULL,
// "otlp_aggregation_temporality" STRING NULL,
// "service_name" STRING NULL,
// "service_version" STRING NULL,
// "session_id" STRING NULL,
// "terminal_type" STRING NULL,
// "user_id" STRING NULL,
// TIME INDEX ("greptime_timestamp"),
// PRIMARY KEY ("job", "model", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// PRIMARY KEY ("job", "model", "otlp_aggregation_temporality", "service_name", "service_version", "session_id", "terminal_type", "user_id")
// )
// ENGINE=metric
// WITH(
@@ -6779,7 +6782,7 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
// on_physical_table = 'greptime_physical_table',
// otlp_metric_compat = 'prom'
// )
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"job\\\", \\\"model\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
let expected = "[[\"claude_code_cost_usage_USD_total\",\"CREATE TABLE IF NOT EXISTS \\\"claude_code_cost_usage_USD_total\\\" (\\n \\\"greptime_timestamp\\\" TIMESTAMP(3) NOT NULL,\\n \\\"greptime_value\\\" DOUBLE NULL,\\n \\\"job\\\" STRING NULL,\\n \\\"model\\\" STRING NULL,\\n \\\"otlp_aggregation_temporality\\\" STRING NULL,\\n \\\"service_name\\\" STRING NULL,\\n \\\"service_version\\\" STRING NULL,\\n \\\"session_id\\\" STRING NULL,\\n \\\"terminal_type\\\" STRING NULL,\\n \\\"user_id\\\" STRING NULL,\\n TIME INDEX (\\\"greptime_timestamp\\\"),\\n PRIMARY KEY (\\\"job\\\", \\\"model\\\", \\\"otlp_aggregation_temporality\\\", \\\"service_name\\\", \\\"service_version\\\", \\\"session_id\\\", \\\"terminal_type\\\", \\\"user_id\\\")\\n)\\n\\nENGINE=metric\\nWITH(\\n 'comment' = 'Created on insertion',\\n 'greptime.semantic.metric.metadata_quality' = 'declared',\\n 'greptime.semantic.metric.original_name' = 'claude_code.cost.usage',\\n 'greptime.semantic.metric.temporality' = 'delta',\\n 'greptime.semantic.metric.type' = 'counter',\\n 'greptime.semantic.metric.unit' = 'USD',\\n 'greptime.semantic.signal_type' = 'metric',\\n 'greptime.semantic.source' = 'opentelemetry',\\n on_physical_table = 'greptime_physical_table',\\n otlp_metric_compat = 'prom'\\n)\"]]";
validate_data(
"otlp_metrics_show_create_table_none",
&client,
@@ -6789,11 +6792,11 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
.await;
// select metrics data
let expected = "[[1753780559836,0.0052544,\"claude-code\",\"claude-3-5-haiku-20241022\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,2.244618,\"claude-code\",\"claude-sonnet-4-20250514\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
let expected = "[[1753780559836,2.244618,\"claude-code\",\"claude-sonnet-4-20250514\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"],[1753780559836,0.0052544,\"claude-code\",\"claude-3-5-haiku-20241022\",\"delta\",\"claude-code\",\"1.0.62\",\"736525A3-F5D4-496B-933E-827AF23A5B97\",\"ghostty\",\"6DA02FD9-B5C5-4E61-9355-9FE8EC9A0CF4\"]]";
validate_data(
"otlp_metrics_select_none",
&client,
"select * from `claude_code_cost_usage_USD_total`;",
"select * from `claude_code_cost_usage_USD_total` order by model desc;",
expected,
)
.await;
@@ -6858,6 +6861,105 @@ pub async fn test_otlp_metrics_new(store_type: StorageType) {
)
.await;
let content = r#"
{"resourceMetrics":[{"scopeMetrics":[{"metrics":[{"name":"http.stale.sum","sum":{"dataPoints":[{"timeUnixNano":"1753780559836000000","flags":1,"asDouble":99.0}],"aggregationTemporality":1,"isMonotonic":true}},{"name":"http.stale.histogram","histogram":{"dataPoints":[{"timeUnixNano":"1753780559835000000","count":"3","sum":4.0,"bucketCounts":["1","1","1"],"explicitBounds":[1.0,2.0]},{"timeUnixNano":"1753780559836000000","flags":1,"count":"99","sum":99.0,"bucketCounts":["99"],"explicitBounds":[1.0,2.0]}],"aggregationTemporality":1}}]}]}]}
"#;
let req: ExportMetricsServiceRequest = serde_json::from_str(content).unwrap();
let res = send_req(
&client,
vec![(
HeaderName::from_static("content-type"),
HeaderValue::from_static("application/x-protobuf"),
)],
"/v1/otlp/v1/metrics",
req.encode_to_vec(),
false,
)
.await;
assert_eq!(StatusCode::OK, res.status());
let body = ExportMetricsServiceResponse::decode(res.bytes().await).unwrap();
assert!(body.partial_success.is_none());
validate_data(
"otlp_http_stale_sum_identity",
&client,
"select otlp_aggregation_temporality, count(*) from http_stale_sum_total \
group by otlp_aggregation_temporality;",
"[[\"delta\",1]]",
)
.await;
for (table, expected) in [
("http_stale_histogram_bucket", "[[\"delta\",6]]"),
("http_stale_histogram_sum", "[[\"delta\",2]]"),
("http_stale_histogram_count", "[[\"delta\",2]]"),
] {
validate_data(
"otlp_http_stale_histogram_identity",
&client,
&format!(
"select otlp_aggregation_temporality, count(*) from {table} \
group by otlp_aggregation_temporality;"
),
expected,
)
.await;
}
let content = r#"
{"resourceMetrics":[{"scopeMetrics":[{"metrics":[{"name":"reserved_label_gauge","gauge":{"dataPoints":[{"timeUnixNano":"1753780559836000000","asDouble":1.0,"attributes":[{"key":"otlp_aggregation_temporality","value":{"stringValue":"user"}}]}]}}]}]}]}
"#;
let req: ExportMetricsServiceRequest = serde_json::from_str(content).unwrap();
let res = send_req(
&client,
vec![(
HeaderName::from_static("content-type"),
HeaderValue::from_static("application/x-protobuf"),
)],
"/v1/otlp/v1/metrics",
req.encode_to_vec(),
false,
)
.await;
assert_eq!(StatusCode::BAD_REQUEST, res.status());
let status = GoogleRpcStatus::decode(res.bytes().await.as_ref()).unwrap();
assert_eq!(tonic::Code::InvalidArgument as i32, status.code);
assert!(status.message.contains("reserved label"));
validate_data(
"otlp_metrics_reserved_label_stores_nothing",
&client,
"select count(*) from information_schema.tables where table_name = 'reserved_label_gauge';",
"[[0]]",
)
.await;
let content = r#"
{"resourceMetrics":[{"scopeMetrics":[{"metrics":[{"name":"malformed_delta_histogram","histogram":{"dataPoints":[{"timeUnixNano":"1753780559836000000","count":"1","sum":1.0,"bucketCounts":["1"],"explicitBounds":[1.0,2.0]}],"aggregationTemporality":1}}]}]}]}
"#;
let req: ExportMetricsServiceRequest = serde_json::from_str(content).unwrap();
let res = send_req(
&client,
vec![(
HeaderName::from_static("content-type"),
HeaderValue::from_static("application/x-protobuf"),
)],
"/v1/otlp/v1/metrics",
req.encode_to_vec(),
false,
)
.await;
assert_eq!(StatusCode::BAD_REQUEST, res.status());
let status = GoogleRpcStatus::decode(res.bytes().await.as_ref()).unwrap();
assert_eq!(tonic::Code::InvalidArgument as i32, status.code);
assert!(status.message.contains("bucket_counts length"));
validate_data(
"otlp_metrics_malformed_delta_histogram_stores_nothing",
&client,
"select count(*) from information_schema.tables where table_name in \
('malformed_delta_histogram_bucket', 'malformed_delta_histogram_sum', \
'malformed_delta_histogram_count');",
"[[0]]",
)
.await;
guard.remove_all().await;
}
tests/cases/standalone/common/promql/delta_temporality.result
+221
View File
@@ -0,0 +1,221 @@
CREATE TABLE delta_temporality (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
series STRING,
otlp_aggregation_temporality STRING,
PRIMARY KEY (series, otlp_aggregation_temporality)
);
Affected Rows: 0
INSERT INTO delta_temporality VALUES
(60000, 10, 'delta', 'delta'),
(120000, 20, 'delta', 'delta'),
(180000, 15, 'delta', 'delta'),
(180000, 7, 'single', 'delta'),
(60000, 10, 'cumulative', NULL),
(120000, 20, 'cumulative', NULL),
(180000, 30, 'cumulative', NULL);
Affected Rows: 7
-- Raw deltas are summed; untagged rows retain cumulative reset-aware math.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') increase(delta_temporality[3m]);
+---------------------+---------------------------------------------------------+------------+------------------------------+
| ts | prom_increase(ts_range,greptime_value,ts,Int64(180000)) | series | otlp_aggregation_temporality |
+---------------------+---------------------------------------------------------+------------+------------------------------+
| 1970-01-01T00:03:00 | 30.0 | cumulative | |
| 1970-01-01T00:03:00 | 45.0 | delta | delta |
| 1970-01-01T00:03:00 | 7.0 | single | delta |
+---------------------+---------------------------------------------------------+------------+------------------------------+
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') rate(delta_temporality[3m]);
+---------------------+-----------------------------------------------------+------------+------------------------------+
| ts | prom_rate(ts_range,greptime_value,ts,Int64(180000)) | series | otlp_aggregation_temporality |
+---------------------+-----------------------------------------------------+------------+------------------------------+
| 1970-01-01T00:03:00 | 0.03888888888888889 | single | delta |
| 1970-01-01T00:03:00 | 0.16666666666666666 | cumulative | |
| 1970-01-01T00:03:00 | 0.25 | delta | delta |
+---------------------+-----------------------------------------------------+------------+------------------------------+
-- The physical plan selects raw-delta math per series while retaining cumulative rate.
-- SQLNESS REPLACE (metrics.*) REDACTED
-- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED
-- SQLNESS REPLACE (-+) -
-- SQLNESS REPLACE (\s\s+) _
-- SQLNESS REPLACE (?m)^\|\s1_\|\s0_\|_(?:Projection|Filter)Exec:.*\n
-- SQLNESS REPLACE (?m)^\|_\|_\|_FilterExec:.*\n
-- SQLNESS REPLACE END\sas\s.*,\sseries@2\sas\sseries END as RATE_RESULT, series@2 as series
-- SQLNESS REPLACE (peers.*) REDACTED
-- SQLNESS REPLACE input_partitions=\d+ input_partitions=REDACTED
-- SQLNESS REPLACE "partition_count":\{(.*?)\} "partition_count":REDACTED
-- SQLNESS REPLACE region=\d+\(\d+,\s+\d+\) region=REDACTED
TQL ANALYZE (180, 180, '1m') rate(delta_temporality[3m]);
+-+-+-+
| stage | node | plan_|
+-+-+-+
| 0_| 0_|_CooperativeExec REDACTED
|_|_|_MergeScanExec: REDACTED
|_|_|_|
|_|_|_ProjectionExec: expr=[ts@0 as ts, CASE WHEN otlp_aggregation_temporality@3 = delta THEN prom_sum_over_time(ts_range@4, greptime_value@1) / 180 ELSE prom_rate(ts_range@4, greptime_value@1, ts@0, 180000) END as RATE_RESULT, series@2 as series, otlp_aggregation_temporality@3 as otlp_aggregation_temporality] REDACTED
|_|_|_PromRangeManipulateExec: req range=[180000..180000], interval=[60000], eval range=[180000], time index=[ts] REDACTED
|_|_|_PromSeriesNormalizeExec: offset=[0], time index=[ts], filter NaN: [true] REDACTED
|_|_|_PromSeriesDivideExec: tags=["series", "otlp_aggregation_temporality"] REDACTED
|_|_|_CooperativeExec REDACTED
|_|_|_SeriesScan: region=REDACTED, "partition_count":REDACTED, "distribution":"PerSeries", "mode":"legacy" REDACTED
|_|_|_|
|_|_| Total rows: 3_|
+-+-+-+
-- The reserved temporality marker treats NULL as the absent cumulative state.
TQL EVAL (180, 180, '1m') delta_temporality{otlp_aggregation_temporality=""};
+---------------------+----------------+------------+------------------------------+
| ts | greptime_value | series | otlp_aggregation_temporality |
+---------------------+----------------+------------+------------------------------+
| 1970-01-01T00:03:00 | 30.0 | cumulative | |
+---------------------+----------------+------------+------------------------------+
TQL EVAL (180, 180, '1m') delta_temporality{otlp_aggregation_temporality!="delta"};
+---------------------+----------------+------------+------------------------------+
| ts | greptime_value | series | otlp_aggregation_temporality |
+---------------------+----------------+------------+------------------------------+
| 1970-01-01T00:03:00 | 30.0 | cumulative | |
+---------------------+----------------+------------+------------------------------+
-- Generated vector plans retain timestamp broadcast across the temporality marker.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') vector(2) * delta_temporality;
+------------+------------------------------+---------------------+----------------------------------------------------+
| series | otlp_aggregation_temporality | ts | .greptime_value * delta_temporality.greptime_value |
+------------+------------------------------+---------------------+----------------------------------------------------+
| cumulative | | 1970-01-01T00:03:00 | 60.0 |
| delta | delta | 1970-01-01T00:03:00 | 30.0 |
| single | delta | 1970-01-01T00:03:00 | 14.0 |
+------------+------------------------------+---------------------+----------------------------------------------------+
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') vector(2) * ignoring(otlp_aggregation_temporality) delta_temporality;
+------------+------------------------------+---------------------+----------------------------------------------------+
| series | otlp_aggregation_temporality | ts | .greptime_value * delta_temporality.greptime_value |
+------------+------------------------------+---------------------+----------------------------------------------------+
| cumulative | | 1970-01-01T00:03:00 | 60.0 |
| delta | delta | 1970-01-01T00:03:00 | 30.0 |
| single | delta | 1970-01-01T00:03:00 | 14.0 |
+------------+------------------------------+---------------------+----------------------------------------------------+
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') delta_temporality * vector(2);
+------------+------------------------------+---------------------+----------------------------------------------------+
| series | otlp_aggregation_temporality | ts | delta_temporality.greptime_value * .greptime_value |
+------------+------------------------------+---------------------+----------------------------------------------------+
| cumulative | | 1970-01-01T00:03:00 | 60.0 |
| delta | delta | 1970-01-01T00:03:00 | 30.0 |
| single | delta | 1970-01-01T00:03:00 | 14.0 |
+------------+------------------------------+---------------------+----------------------------------------------------+
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') delta_temporality * ignoring(otlp_aggregation_temporality) vector(2);
+------------+------------------------------+---------------------+----------------------------------------------------+
| series | otlp_aggregation_temporality | ts | delta_temporality.greptime_value * .greptime_value |
+------------+------------------------------+---------------------+----------------------------------------------------+
| cumulative | | 1970-01-01T00:03:00 | 60.0 |
| delta | delta | 1970-01-01T00:03:00 | 30.0 |
| single | delta | 1970-01-01T00:03:00 | 14.0 |
+------------+------------------------------+---------------------+----------------------------------------------------+
-- Aggregation preserves or deliberately removes the visible stored marker.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') sum by (series, otlp_aggregation_temporality) (rate(delta_temporality[3m]));
+------------+------------------------------+---------------------+----------------------------------------------------------+
| series | otlp_aggregation_temporality | ts | sum(prom_rate(ts_range,greptime_value,ts,Int64(180000))) |
+------------+------------------------------+---------------------+----------------------------------------------------------+
| cumulative | | 1970-01-01T00:03:00 | 0.16666666666666666 |
| delta | delta | 1970-01-01T00:03:00 | 0.25 |
| single | delta | 1970-01-01T00:03:00 | 0.03888888888888889 |
+------------+------------------------------+---------------------+----------------------------------------------------------+
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') sum by (series) (rate(delta_temporality[3m]));
+------------+---------------------+----------------------------------------------------------+
| series | ts | sum(prom_rate(ts_range,greptime_value,ts,Int64(180000))) |
+------------+---------------------+----------------------------------------------------------+
| cumulative | 1970-01-01T00:03:00 | 0.16666666666666666 |
| delta | 1970-01-01T00:03:00 | 0.25 |
| single | 1970-01-01T00:03:00 | 0.03888888888888889 |
+------------+---------------------+----------------------------------------------------------+
TQL EVAL (180, 180, '1m') round(sum(rate(delta_temporality[3m])), 0.000001);
+---------------------+----------------------------------------------------------------------------------------+
| ts | prom_round(sum(prom_rate(ts_range,greptime_value,ts,Int64(180000))),Float64(0.000001)) |
+---------------------+----------------------------------------------------------------------------------------+
| 1970-01-01T00:03:00 | 0.45555599999999996 |
+---------------------+----------------------------------------------------------------------------------------+
CREATE TABLE delta_marker_only (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
otlp_aggregation_temporality STRING PRIMARY KEY
);
Affected Rows: 0
INSERT INTO delta_marker_only VALUES
(180000, 1, 'delta'),
(180000, 2, NULL);
Affected Rows: 2
CREATE TABLE delta_tagless (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE
);
Affected Rows: 0
INSERT INTO delta_tagless VALUES (180000, 10);
Affected Rows: 1
-- A missing temporality marker matches the NULL cumulative state; "delta" does not.
TQL EVAL (180, 180, '1m') delta_marker_only + delta_tagless;
+---------------------+-----------------------------------------------------------------+
| ts | delta_marker_only.greptime_value + delta_tagless.greptime_value |
+---------------------+-----------------------------------------------------------------+
| 1970-01-01T00:03:00 | 12.0 |
+---------------------+-----------------------------------------------------------------+
TQL EVAL (180, 180, '1m') delta_marker_only AND delta_tagless;
+---------------------+----------------+------------------------------+
| ts | greptime_value | otlp_aggregation_temporality |
+---------------------+----------------+------------------------------+
| 1970-01-01T00:03:00 | 2.0 | |
+---------------------+----------------+------------------------------+
DROP TABLE delta_tagless;
Affected Rows: 0
DROP TABLE delta_marker_only;
Affected Rows: 0
DROP TABLE delta_temporality;
Affected Rows: 0
tests/cases/standalone/common/promql/delta_temporality.sql
+83
View File
@@ -0,0 +1,83 @@
CREATE TABLE delta_temporality (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
series STRING,
otlp_aggregation_temporality STRING,
PRIMARY KEY (series, otlp_aggregation_temporality)
);
INSERT INTO delta_temporality VALUES
(60000, 10, 'delta', 'delta'),
(120000, 20, 'delta', 'delta'),
(180000, 15, 'delta', 'delta'),
(180000, 7, 'single', 'delta'),
(60000, 10, 'cumulative', NULL),
(120000, 20, 'cumulative', NULL),
(180000, 30, 'cumulative', NULL);
-- Raw deltas are summed; untagged rows retain cumulative reset-aware math.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') increase(delta_temporality[3m]);
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') rate(delta_temporality[3m]);
-- The physical plan selects raw-delta math per series while retaining cumulative rate.
-- SQLNESS REPLACE (metrics.*) REDACTED
-- SQLNESS REPLACE (RoundRobinBatch.*) REDACTED
-- SQLNESS REPLACE (-+) -
-- SQLNESS REPLACE (\s\s+) _
-- SQLNESS REPLACE (?m)^\|\s1_\|\s0_\|_(?:Projection|Filter)Exec:.*\n
-- SQLNESS REPLACE (?m)^\|_\|_\|_FilterExec:.*\n
-- SQLNESS REPLACE END\sas\s.*,\sseries@2\sas\sseries END as RATE_RESULT, series@2 as series
-- SQLNESS REPLACE (peers.*) REDACTED
-- SQLNESS REPLACE input_partitions=\d+ input_partitions=REDACTED
-- SQLNESS REPLACE "partition_count":\{(.*?)\} "partition_count":REDACTED
-- SQLNESS REPLACE region=\d+\(\d+,\s+\d+\) region=REDACTED
TQL ANALYZE (180, 180, '1m') rate(delta_temporality[3m]);
-- The reserved temporality marker treats NULL as the absent cumulative state.
TQL EVAL (180, 180, '1m') delta_temporality{otlp_aggregation_temporality=""};
TQL EVAL (180, 180, '1m') delta_temporality{otlp_aggregation_temporality!="delta"};
-- Generated vector plans retain timestamp broadcast across the temporality marker.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') vector(2) * delta_temporality;
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') vector(2) * ignoring(otlp_aggregation_temporality) delta_temporality;
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') delta_temporality * vector(2);
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') delta_temporality * ignoring(otlp_aggregation_temporality) vector(2);
-- Aggregation preserves or deliberately removes the visible stored marker.
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') sum by (series, otlp_aggregation_temporality) (rate(delta_temporality[3m]));
-- SQLNESS SORT_RESULT 3 1
TQL EVAL (180, 180, '1m') sum by (series) (rate(delta_temporality[3m]));
TQL EVAL (180, 180, '1m') round(sum(rate(delta_temporality[3m])), 0.000001);
CREATE TABLE delta_marker_only (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE,
otlp_aggregation_temporality STRING PRIMARY KEY
);
INSERT INTO delta_marker_only VALUES
(180000, 1, 'delta'),
(180000, 2, NULL);
CREATE TABLE delta_tagless (
ts TIMESTAMP TIME INDEX,
greptime_value DOUBLE
);
INSERT INTO delta_tagless VALUES (180000, 10);
-- A missing temporality marker matches the NULL cumulative state; "delta" does not.
TQL EVAL (180, 180, '1m') delta_marker_only + delta_tagless;
TQL EVAL (180, 180, '1m') delta_marker_only AND delta_tagless;
DROP TABLE delta_tagless;
DROP TABLE delta_marker_only;
DROP TABLE delta_temporality;
tests/perf/query_cases/prom_remote_write_seeded_random/case.toml
+10
View File
@@ -46,3 +46,13 @@ iterations = 15
[scenario.queries.thresholds]
max_candidate_latency_regression_pct = 75
[[scenario.queries]]
name = "binary_1d_ignoring_host"
kind = "tql"
query = "TQL ANALYZE VERBOSE (1704067200, 1704153600, '1h') prom_remote_write_seeded_random / ignoring(host) prom_remote_write_seeded_random"
warmup = 2
iterations = 9
[scenario.queries.thresholds]
max_candidate_latency_regression_pct = 75