mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
Loakesh Indiran
c679698fa5
## Problem `ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave wrong results and risked OOM because per-partition \`hits_to_skip\` assumed rows are time-ordered — but the globally k-th row can be in any partition. ## Solution Each partition fetches its local top-`k = from+size` rows. A **TopKHeap** on the leader merges them globally, skips `from`, returns `size`. Heap is bounded to `k` elements at all times — O(k) memory regardless of partition count. **Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is fully honored: heap comparison and final sort respect all columns in order, using secondary columns as tiebreakers. ## Scenario behavior | Query | Behavior | |---|---| | `ORDER BY _timestamp DESC` | Unchanged — per-partition path | | `ORDER BY duration DESC` | Heap merge path — correct pagination | | `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in `duration` broken by `_timestamp` | | `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips `from`, returns `size` | | JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition (unchanged) | | `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition (circuit breaker preserved) | | Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no regression | ## What is NOT changed - `ORDER BY _timestamp` execution path - `_search` (non-streaming) — DataFusion handles it correctly - Aggregate / histogram queries