mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 08:09:24 +00:00
ByteBaker
ada80858e8
Replace in-memory EdgeStore with daemon-based architecture that periodically
queries trace streams using SQL aggregation in DataFusion.
Architecture:
- Daemon runs in compactor job every 1 hour (default, configurable)
- Queries last 60 minutes of trace data (configurable window)
- Uses DataFusion SQL with CTE for efficient aggregation
- Calculates p50, p95, p99 percentiles using approx functions
- Writes aggregated edges to internal `_o2_service_graph` stream
- Zero impact on trace ingestion performance
Key Changes:
- Add `processor.rs` with SQL-based daemon that queries trace streams
- Add `aggregator.rs` to write SQL results to `_o2_service_graph` stream (uses json! macro)
- Refactor `api.rs` to query pre-aggregated data from stream
- Move business logic to enterprise repo (`build_topology()`, `span_to_graph_span()`)
- Remove ALL inline processing during trace ingestion
- Delete in-memory EdgeStore, worker threads, and buffering code
- Feature-gated for enterprise builds with non-enterprise stubs
SQL Query:
- CTE extracts client/server from span_kind ('2'=SERVER, '3'=CLIENT)
- Aggregates by (client, server) with COUNT, percentiles, error rate
- Filters: span_kind IN ('2','3') AND peer_service IS NOT NULL
Configuration:
- O2_SERVICE_GRAPH_ENABLED (default: false)
- O2_SERVICE_GRAPH_PROCESSING_INTERVAL_SECS (default: 3600) - Daemon run frequency
- O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES (default: 60) - Query window size
Technical Details:
- Stream name: `_o2_service_graph` (internal stream)
- Edge timestamp: Uses query end time for API compatibility
- Span kinds: '2' (SERVER), '3' (CLIENT) as VARCHAR in parquet
- Percentiles: approx_median(), approx_percentile_cont(0.95), approx_percentile_cont(0.99)