mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
fix: snap histogram scan start down to bucket boundary to fix left-edge drop (#12313)
## Problem Histogram panels showed a false drop in the **leftmost bucket**. `histogram()` rewrites to `date_bin(interval, _timestamp, epoch)`, which assigns each record to the **left boundary** of its bucket. So a record at `08:10:17` gets key `08:10:00`. But the scan filter was `_timestamp >= start_time` where `start_time` was the raw query time (e.g. `08:10:17`). Records in `[08:10:00, 08:10:17)` passed the bucket label check but were excluded from the scan — making the first bucket appear partially filled. Fixes #1753 (o2-enterprise) ## Fix Snap `start_time` **down to the nearest bucket boundary** at the **leader node**, before it is broadcast to all follower nodes via `FlightSearchRequest`. Location: `RemoteScanNodes::get_remote_node()` in `distributed_plan/node.rs` ```rust start_time: { let t = self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0); if self.req.histogram_interval > 0 { histogram_bucket_start(t, self.req.histogram_interval * 1_000_000) } else { t } }, ``` The arithmetic is extracted into `histogram_bucket_start(timestamp_us, interval_us)` in `sql/histogram.rs` so the formula lives in one place. Doing the snap at the leader means: - All followers receive an already-aligned `start_time` - `SimpleHistogramVisitor` / `SimpleMultiHistogramVisitor` on followers need no changes (their `start_time % rounding_by` is already 0) - Covers both cluster leader and super cluster leader (both go through `generate_remote_scan_rules → get_remote_node()`) ## Test plan - [ ] Deploy and run a histogram query spanning a non-aligned `start_time` - [ ] Verify first bucket count matches adjacent full buckets - [ ] Verify queries without `histogram_interval` are unaffected
This commit is contained in:
@@ -23,6 +23,8 @@ use datafusion::common::TableReference;
|
||||
use hashbrown::HashMap;
|
||||
use proto::cluster_rpc::{IndexInfo, KvItem, QueryIdentifier, SearchInfo, SuperClusterInfo};
|
||||
|
||||
use crate::service::search::sql::histogram::histogram_bucket_start;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RemoteScanNodes {
|
||||
pub req: Request,
|
||||
@@ -73,7 +75,14 @@ impl RemoteScanNodes {
|
||||
.get(table_name)
|
||||
.unwrap_or(&vec![])
|
||||
.clone(),
|
||||
start_time: self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0),
|
||||
start_time: {
|
||||
let t = self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0);
|
||||
if self.req.histogram_interval > 0 {
|
||||
histogram_bucket_start(t, self.req.histogram_interval * 1_000_000)
|
||||
} else {
|
||||
t
|
||||
}
|
||||
},
|
||||
end_time: self.req.time_range.as_ref().map(|x| x.1).unwrap_or(0),
|
||||
timeout: self.req.timeout as u64,
|
||||
use_cache: self.req.use_cache,
|
||||
|
||||
@@ -190,6 +190,13 @@ fn extract_where_clause(statement: &Statement) -> Result<String, Error> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Snaps `timestamp_us` down to the nearest histogram bucket boundary.
|
||||
/// Both arguments are in microseconds. Used to ensure the first bucket
|
||||
/// returned by date_bin() is fully populated rather than partial.
|
||||
pub fn histogram_bucket_start(timestamp_us: i64, interval_us: i64) -> i64 {
|
||||
timestamp_us - timestamp_us % interval_us
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
Reference in New Issue
Block a user