---
Feature Name: Native Histogram Support and Compatibility Decisions
Tracking Issue: TBD
Date: 2026-08-04
Author: codex
---

# Summary

GreptimeDB stores native histograms in one Prometheus-compatible Struct-valued
field and evaluates them as first-class PromQL samples. Prometheus Remote Write
2.0 supplies native histograms directly. OTLP `ExponentialHistogram` is an
ingestion transport: accepted cumulative points are normalized into the same
Struct before persistence and are queried only as native histograms.

Native histograms are experimental. Prometheus Remote Write 2.0 and cumulative
OTLP exponential histograms are supported behind separate configuration gates.
Remote Write 1.0 histogram payloads are rejected instead of being acknowledged and dropped.
Native-histogram Remote Read is deferred; the existing Remote Read path
continues to return scalar samples only.

# Goals

1. Prevent silent native-histogram data loss at protocol boundaries.
2. Keep one stable Prometheus-compatible persisted representation across
   Prometheus and OTLP ingestion.
3. Match Prometheus query behavior where it is observable and practical.
4. State intentional limitations explicitly so incomplete behavior is not
   mistaken for support.

# Non-Goals

- Supporting native histograms in Remote Write 1.0.
- Returning native histograms through Prometheus Remote Read.
- Persisting Remote Write metric metadata.
- Persisting exemplars.
- Providing an OTLP-specific histogram query surface or reconstructing and
  re-exporting the original OTLP point after persistence.
- Removing mixed float/histogram handling from PromQL expressions.
- Propagating PromQL annotations produced on datanodes back to the frontend.

# Data Model and Invariants

Each accepted native histogram, whether received directly through Remote Write
2.0 or normalized from OTLP `ExponentialHistogram`, is stored in the configured
native-histogram field (`greptime_native_histogram` by default) as the same
canonical Struct. The persistence boundary admits only values accepted by the
shared native-histogram validation.

## Prometheus Remote Write 2.0

Remote Write 2.0 native histograms enter the persistence path in Prometheus
format. The Struct preserves the validated schema, zero threshold, sum, reset
hint, start timestamp, custom bounds, spans, and either the integer or float
count family. Integer bucket deltas are converted to absolute integer counts
for storage; float bucket counts are already absolute. No separate sample-kind
discriminator is stored because the populated count family identifies it.

## OTLP ExponentialHistogram

OTLP `ExponentialHistogram` does not introduce a second persisted format.
Accepted cumulative points are normalized into a valid Prometheus native
histogram and then pass through the same validator and Struct encoder. The
Struct does not retain the raw OTLP point or a source-protocol discriminator,
and queries use only the native-histogram PromQL behavior. Detailed conversion
and rejection rules are listed under [OTLP](#otlp).

Within one resolved catalog, schema, and physical-table routing context, a
metric name has exactly one persisted sample kind:

- a float metric uses `greptime_value`;
- a native-histogram metric uses the configured native-histogram field.

Labels do not change that choice. Remote Write 2.0 rejects a request when two
label sets use different sample kinds for the same metric, and the existing
table schema rejects kind changes across requests. Prometheus metadata type is
not used for this decision: a classic histogram family is described as a
histogram but is represented by float-valued `_bucket`, `_sum`, and `_count`
series.

The storage invariant does not remove the need for mixed-sample PromQL support.
An expression can combine different metrics or branches, for example with
`or`, and therefore produce a vector containing float samples and histogram
samples from different series.

# Protocol Decisions

## Remote Write 1.0

Remote Write 1.0 scalar samples remain supported. Any `TimeSeries.histograms`
field causes the complete request to fail with an invalid-arguments response.
Rejecting at protobuf decode time prevents both histogram-only data loss and
partial ingestion of a request that mixes scalar and histogram series. Clients
that send native histograms must use Remote Write 2.0.

## Remote Write 2.0

Remote Write 2.0 accepts integer and float native histograms while
`prom_store.experimental_enable_prometheus_native_histogram` is enabled. Supported
exponential schemas are `-4` through `8`; schema `-53` represents native
histograms with custom buckets.

Exponential spans may end at the schema's overflow bucket but must not continue
beyond it. The overflow index is `(1024 << schema) + 1` for non-negative schemas
and `(1024 >> -schema) + 1` for negative schemas. Inputs beyond that index are
rejected rather than clamped or merged into infinity.

Exemplars and declared metric metadata are accepted on the wire but are not
persisted by this feature.

## Remote Read

Native-histogram Remote Read is deferred. GreptimeDB's existing Remote Read path
continues to negotiate `SAMPLES` responses and serialize scalar samples only.

A follow-up can map the stored Struct into `TimeSeries.histograms`. It must
preserve integer counts without passing through `f64`, retain the stored
histogram fields, and decide sampled versus streamed response coverage. The
sampled protobuf has no start-timestamp field, so that omission must remain
explicit if sampled responses are implemented.

## OTLP

OTLP exponential histograms are accepted when
`otlp.experimental_enable_exponential_histogram` is enabled. The option defaults
to false and applies to OTLP/HTTP. Disabled points are rejected rather than
silently acknowledged. OTel Arrow exponential histograms are rejected because
the current Arrow wire format omits `zero_threshold`; accepting them would
silently change the distribution. Cumulative temporality is required; delta and
unspecified exponential histograms are rejected before their points are
converted. Explicit OTLP histograms keep their existing `_bucket`, `_sum`, and
`_count` representation, including their existing delta behavior.

OTLP scales `-4` through `8` map directly to Prometheus schemas. Higher scales
are downscaled to schema `8`: dense counts that collide are merged before the
OTLP lower-bound index is shifted by one to the Prometheus upper-bound index.
This preserves count mass but irreversibly loses distinctions between source
buckets that merge. Lower scales are rejected. OTLP counts always populate the
integer histogram family. The transmitted non-negative finite zero threshold
and zero count are preserved; the Struct start timestamp is stored in
milliseconds and the reset hint is unknown. Point timestamps and attributes
retain their existing mode-specific conversion rules. Legacy mode retains its
normalized name, attribute rules, and nanosecond row timestamp. Non-legacy mode
retains Prometheus-compatible translation and its millisecond row timestamp.
Both modes write the same canonical Struct.

An absent sum is stored as an ordinary quiet NaN, so the sample remains
selectable while `histogram_sum` is unknown. Any NaN sum on a point without
`NoRecordedValue` is normalized to the same value, even if its payload has the
Prometheus stale-marker bits. Only an exponential-histogram point carrying OTLP
`NoRecordedValue` becomes an empty schema-0 integer histogram with the canonical
Prometheus stale-NaN sum; its attributes and timestamps remain. This
interpretation intentionally does not change gauges, sums, or explicit histograms.

Invalid points are skipped while unrelated valid points continue. Mixed
accepted/rejected OTLP/HTTP requests return partial success; a request with only
rejected points returns `InvalidArgument`. OTel Arrow uses an `OK` batch status
for mixed batches and `INVALID_ARGUMENT` when all points are rejected. Rejection
details are bounded, and metric metadata is emitted only for a metric that
produced an accepted row.

Minimum, maximum, and exemplars are not persisted. Delta accumulation,
zero-run span compaction, and a dedicated rejection metric remain deferred.

# PromQL Compatibility Decisions

## Staleness

A native histogram is stale only when its `sum` has Prometheus's stale-NaN bit
pattern. Ordinary NaN sums remain samples. Range selectors remove stale markers
before range functions run. Instant selectors treat the marker as the end of
the series for lookback purposes and do not resurrect an older sample.

## Start timestamps

Reset detection already uses consecutive start timestamps in addition to
bucket and count monotonicity. Counter `rate` and `increase` also use the first
histogram's start timestamp as a synthetic zero when it is nonzero and strictly
inside the query window before the first sample. This permits the same
single-sample calculation as Prometheus and prevents left extrapolation past the
known start. When no usable start timestamp exists, a positive histogram-count
increase lets counter extrapolation infer a zero point from the first count to
cap left extrapolation; a synthetic zero uses its known timestamp instead of
that heuristic.

This behavior is native-histogram-only. GreptimeDB does not persist start
timestamps for float samples, so float `rate` and `increase` still require two
samples. Warnings for overlapping start timestamps are deferred.

## Arithmetic and averages

Histogram addition and subtraction always reconcile layouts, including empty
histograms. Mixing exponential and custom layouts fails; custom layouts are
reconciled according to their shared bounds. The only layout bypass is local to
counter rate: when the first-to-second pair is a reset, the first sample's
layout is irrelevant and the second sample starts the accumulated segment.

`avg` and `avg_over_time` use a mergeable weighted running mean. This prevents a
finite mean from becoming infinity solely because the intermediate sum
overflowed. Native-histogram aggregates are not currently split into datanode
partial aggregation and frontend state merging, so this state does not cross
nodes or versions. If distributed stepping is added later, its state must be
versioned before rolling upgrades can safely mix implementations.

The rejected alternative is Prometheus's direct-sum/Kahan-compensation design
with an overflow-triggered mode change. It offers closer rounding parity but
requires additional state and substantially more merge logic.

## Equality, changes, and resets

Equality follows Prometheus's represented-layout semantics. Histograms compare
equal only when their represented bucket-index sequences and bucket value bit
patterns match. An explicitly represented zero bucket therefore differs from an
omitted bucket, and `changes()` observes that layout change. Redundant
zero-length span encodings are normalized, malformed layouts compare unequal,
and reset hints and start timestamps remain excluded.

Custom bucket bounds use ordinary floating-point equality rather than bitwise
equality, matching Prometheus's `CustomBucketBoundsMatch`. Histogram payload
values continue to use bitwise equality so NaN payloads behave deterministically.

`resets()` counts ordinary counter resets and, unlike Prometheus, also counts
each transition between gauge and non-gauge native histograms as one reset.

## Annotations

Warnings and infos produced by frontend-executed PromQL functions are returned
through the corresponding Prometheus JSON response fields. Annotations produced
while a pushed-down plan executes on a datanode are not yet transported back to
the frontend. The sample-dropping behavior is unchanged, but those remote
annotations remain silent until a query-result transport is defined for them.

## Standard deviation and bucket bounds

Custom-bucket standard deviation and variance use the arithmetic midpoint
`(lower + upper) / 2` for every bucket. Underflow and overflow buckets therefore
naturally produce infinite or NaN estimates. Exponential buckets retain the
signed geometric midpoint, with zero used for a bucket spanning zero.

Boundary calculation returns the last finite boundary as `f64::MAX`, the
overflow boundary as positive infinity, and rejects indices beyond the
overflow bucket. Widened integer arithmetic prevents large negative-schema
indices from wrapping into an unrelated finite boundary.

# Metric Metadata

The Prometheus `/metadata` endpoint currently derives metric names from logical
table names and reads type and unit from semantic table options stamped at table
creation. Native-histogram tables without a declared type fall back to
`histogram`. Help is always empty, and unit is absent for tables without a
semantic unit option.

Persisting declared Remote Write metadata is separate work. Reusing table
options alone is insufficient because metadata-only writes, later updates, and
existing tables do not pass through auto-create. An in-memory registry would
lose data on restart. The recommended follow-up is a persistent registry keyed
by catalog, schema, and metric family, populated by Remote Write 1.0 metadata-only
requests and Remote Write 2.0 per-series metadata. This follows the update
boundary already recorded in
[Table Semantic Layer](2026-05-28-table-semantic-layer.md).

# Testing and Compatibility

Compatibility coverage includes protocol rejection, OTLP scale conversion and
partial-success behavior, kind exclusivity,
stale-marker selector semantics, synthetic-zero rates, incompatible empty
layouts, overflow-safe averages, layout-sensitive equality, infinite custom
midpoints, and exponential overflow indices for schemas `-4`, `0`, and `8`.
Behavioral coverage also verifies frontend PromQL warning and info responses.

This work does not change GreptimeDB's persisted Struct or protobuf
dependencies. It adds the disabled-by-default public OTLP gate; existing
native-histogram data remains readable.

# Future Work

- Native-histogram Remote Read, including exact integer round-trips and streamed
  chunks with start timestamps.
- OTLP exponential-histogram delta accumulation.
- Persistent Remote Write metadata and accurate help/unit updates.
- Native-histogram exemplars and exemplar query APIs.
- Start-timestamp overlap annotations.
- Datanode-to-frontend PromQL annotation propagation.
- Versioned two-phase native-histogram aggregation state.
- Exact Prometheus summation compensation if measured precision differences
  justify the extra aggregate state.
