mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
Hengfei Yang
9c82beeeed
Close #12345 ## Problem OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with `encoding: json`) created streams but stored 0 events, with no `_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were silently recorded as 0. The protobuf path was unaffected, which is why this only surfaced for collectors explicitly configured with JSON encoding. ## Root cause `otlp_json` deserializes the body directly into opentelemetry-proto's serde-derived types, which only partially implement the proto3 JSON mapping (64-bit integers encoded as strings, enums by name, zero/empty fields omitted). `Metric.data` is a flattened `Option` oneof, and serde converts any inner deserialization error of a flattened `Option` into `None` instead of failing — so the whole metric silently became `data: None`, while the stream itself had already been created by the schema setup. Same mechanism on `NumberDataPoint.value` turned string `asInt` values into 0. ## Changes - Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded u64 handling for plain histograms and summaries upstream. - Add `otlp_json_compat` normalizer for the gaps still present in 0.32 (each verified empirically, with unit tests): - string `asInt` values on data points and exemplars - `aggregationTemporality` as enum name - `Exemplar`'s value oneof is not flattened in 0.32 - missing `serde(default)` on `ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`, which rejected omitted zero/empty fields - Count and log metrics whose data oneof failed to decode (`partial_success.rejected_data_points`) instead of silently skipping them, and stop overwriting specific partial-success error messages with the generic retention message. - Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`, `AnyValue::StringValueStrindex`). - Upgrade workspace reqwest 0.12 → 0.13, required because opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13 (`rustls` feature replaces `rustls-tls-native-roots` via the platform verifier; `json`/`query`/`form` are now explicit features). The `segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved into `telemetry.rs` as a plain `reqwest::Client` that posts the Track message itself, replicating `segment::HttpClient::send`. - Remove the unused pyroscope profiling integration (optional feature, dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config). ## Verification - 10 unit tests in `otlp_json_compat` covering the full payload matrix (collector-style string integers, enum names, omitted zero fields, exemplars with and without trace context, already-normalized payloads, malformed input). - End-to-end: posted the issue's collector-style JSON payload to `/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative counts with exemplars attached), `_count`, `_sum` populated for histogram and exponential histogram, summary quantiles stored, `asInt` counter stored as 42 instead of 0. - `cargo clippy -- -D warnings` clean; lib test failures are pre-existing local-environment issues that reproduce identically on main.