mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 08:09:24 +00:00
Hengfei Yang
72f7bff378
## Summary - Page sessions by `session_id` before computing the expensive summary fields. - Aggregate only the selected page of sessions with exact `count(DISTINCT trace_id)`, removing trace ID materialization and Rust-side per-trace rollups. - Generate direct Agent predicates and evaluate them as aggregate membership conditions without a session-membership subquery. - Support efficient server pagination by fetching one extra session to determine `has_more`, without counting every distinct session on every page. ## Why The previous endpoint first materialized `array_agg(DISTINCT trace_id)` for every candidate session and then issued another query using those trace IDs. A single-pass full rollup was also slower because it aggregated every summary field for all matching sessions before applying TopK. This change keeps the inexpensive session page selection separate and bounds the expensive rollup to the requested page size. Page selection and display both use the latest `end_time` across the complete session; Agent predicates only decide whether a session is a member of the result set. The endpoint previously returned `total = hits.len()`. With a 20-row page, OTable therefore calculated one total page and permanently disabled Next even when more sessions existed. An exact `count(DISTINCT session_id)` query scanned the full data set and took about 1.18 seconds by itself on this benchmark. Instead, the page query now requests `size + 1`: while another page exists, `total` is a documented lower bound and the response includes `has_more=true` / `total_is_exact=false`. The final page returns the exact total. OTable displays lower bounds with `+` and hides Last-page navigation until the exact total is known. ## Performance Measured locally against `bench_traces` (167 files, about 99.9 million records), using five runs and ignoring the first two: | Query shape | Mean wall time | | --- | ---: | | Original trace-ID flow | 6.829 s | | Page session IDs + membership subquery | 3.219 s | | Page session IDs + Last activity | 0.850 s | The optimized query shape is about 87.6% faster than the original flow. The second-phase rollup returned 20 rows with about 74 KB of intermediate output and about 2.45 MB peak memory in the query plan. Fetching the extra pagination sentinel did not introduce a meaningful page-size-dependent regression in the final runtime sweep. ## Behavior note The Agent filter controls membership through `HAVING max(CASE WHEN <filter> THEN 1 ELSE 0 END) = 1`, while the final rollup includes all spans for each selected session. Both pagination and display use the complete session's `max(end_time)`, ordered by `session_last_activity DESC, session_id DESC` for stable offset pagination. The UI labels this value `Last activity`; session start and duration remain available from the rollup. Current clients send direct Agent predicates. For compatibility, the endpoint also recognizes the previous narrow same-stream `session_id IN (SELECT session_id ... GROUP BY session_id)` shape and safely extracts its inner WHERE predicate; unrelated subqueries are left unchanged. This assumes every relevant span has a `session_id`. ## Validation - `cargo build --features mimalloc --profile release` - `cargo test -p openobserve-api-search traces::session --lib` (18 passed) - `cargo test -p openobserve-api-search user::tests --lib` (13 passed in the earlier query-change validation) - `cargo fmt --all` - `git diff --check` - targeted Vitest: Sessions composable, SessionsList, and OTable (169 passed) - `vue-tsc --noEmit -p tsconfig.vitest.json --composite false` - targeted ESLint and Prettier checks - runtime pagination check against the supplied legacy-filter curl: - page 1: 20 hits, `total=21`, `has_more=true`, `total_is_exact=false` - page 2: 20 hits, `total=41`, `has_more=true`, `total_is_exact=false` - final page: 3 hits, `total=184443`, `has_more=false`, `total_is_exact=true` - pages 1 and 2 returned 40 unique sessions in descending `end_time` order across the page boundary