mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Hengfei Yang
d996349aa4
Design at: #12620 Closes #12620 Fixes #12564 ## What Rewrites both tantivy histogram optimize-rule collectors following the same approach as the recent `TopNCollector` / `SingleSegmentDocIdCollector` rewrites, and adds timezone support to both rules. - `MultiHistogramCollector` (replaces the aggregation framework for `SimpleMultiHistogram`) - `SimpleHistogramCollector` (replaces `tantivy::collector::HistogramCollector` for `SimpleHistogram`) ### Timezone support (#12564) `histogram(ts, interval, tz)` rewrites to `date_bin(interval, to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's `get_column_name` hit the `BinaryExpr` and silently skipped both rules. Now: - the visitors extract the fixed offset from that shape (`get_timestamp_offset`), so the rules fire with a timezone - `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a `ts_offset` field (no proto change — histogram modes never cross gRPC, they're re-derived per node) - bucket edges are computed in shifted wall-clock space and the collectors bucket on `_timestamp + ts_offset`, so emitted keys match `date_bin` exactly when merging with non-tantivy partials ## Testing - New collector unit tests: dense vs sparse equality, ts_offset shifting, bounds filtering, missing column / missing breakdown docs, per-bucket truncation with tie-breaks - Visitor tests now include real-DataFusion-plan timezone cases (`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted bounds and offset - `cargo test -p config` (1936 passed), plus `index_optimizer` / `search::tantivy` / `tantivy_optimize_exec` suites in the main crate; clippy clean - Low cardinality: fast `20%` than main ``` histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC ``` - Low cardinality: fast `30%` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ``` - High cardinality: faster `5x` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ```