- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
Design at: #12620Closes#12620Fixes#12564
## What
Rewrites both tantivy histogram optimize-rule collectors following the
same approach as the recent `TopNCollector` /
`SingleSegmentDocIdCollector` rewrites, and adds timezone support to
both rules.
- `MultiHistogramCollector` (replaces the aggregation framework for
`SimpleMultiHistogram`)
- `SimpleHistogramCollector` (replaces
`tantivy::collector::HistogramCollector` for `SimpleHistogram`)
### Timezone support (#12564)
`histogram(ts, interval, tz)` rewrites to `date_bin(interval,
to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's
`get_column_name` hit the `BinaryExpr` and silently skipped both rules.
Now:
- the visitors extract the fixed offset from that shape
(`get_timestamp_offset`), so the rules fire with a timezone
- `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a
`ts_offset` field (no proto change — histogram modes never cross gRPC,
they're re-derived per node)
- bucket edges are computed in shifted wall-clock space and the
collectors bucket on `_timestamp + ts_offset`, so emitted keys match
`date_bin` exactly when merging with non-tantivy partials
## Testing
- New collector unit tests: dense vs sparse equality, ts_offset
shifting, bounds filtering, missing column / missing breakdown docs,
per-bucket truncation with tie-breaks
- Visitor tests now include real-DataFusion-plan timezone cases
(`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted
bounds and offset
- `cargo test -p config` (1936 passed), plus `index_optimizer` /
`search::tantivy` / `tantivy_optimize_exec` suites in the main crate;
clippy clean
- Low cardinality: fast `20%` than main
```
histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC
```
- Low cardinality: fast `30%` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
- High cardinality: faster `5x` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
Huaijin
Hengfei Yang
Harsh Mahajan
Don Kendall
dependabot[bot]
漂流
udev
