mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 08:09:24 +00:00
sktrpathiandsai nikhil kethe
e52d000217
## Summary Introduces comprehensive incident management system to reduce alert fatigue through intelligent correlation and AI-powered root cause analysis. **Design at:** https://github.com/openobserve/designs/tree/main/alert-correlation --- ## Incident Correlation **3-Tier Correlation Priority:** 1. **Trace-based**: Groups alerts from same distributed trace (cross-service) 2. **Service Discovery**: Uses semantic service matching with normalization 3. **Dimension-based**: Blake3 hash from stable alert dimensions (fallback) **Features:** - Automatic alert grouping into incidents - Real-time incident status tracking (open/acknowledged/resolved/suppressed) - Topology enrichment via Service Graph integration - Cross-environment correlation prevention via dimension matching ## Incident Analysis (RCA) **Automated Analysis:** - Background job queries open incidents and triggers AI agent - Periodic analysis via configurable interval - Results displayed in incident detail view with markdown support **On-Demand Analysis:** - Manual trigger via API: `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Real-time results with loading indicators - One-click "Analyze" button in UI ## SRE Chat Assistant **Conversational AI for Alerts:** - Generic `/api/{org}/agent/chat` endpoint with history support - Context-aware assistance for alerts and incidents - Sparkle icons on alert/incident rows for quick access - Persistent chat history via IndexedDB - Includes RCA analysis in conversation context --- ## Configuration ```bash # Incident Management O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES=-1 # -1=disabled (default), >0=auto-resolve # RCA Agent Integration O2_INCIDENTS_RCA_ENABLED=true O2_INCIDENTS_RCA_AGENT_URL=http://localhost:8000 O2_INCIDENTS_RCA_INTERVAL_SECS=600 ``` ## API Endpoints - `GET /api/v2/{org}/alerts/incidents` - List incidents with pagination - `GET /api/v2/{org}/alerts/incidents/{id}` - Get incident details - `PATCH /api/v2/{org}/alerts/incidents/{id}` - Update incident status - `GET /api/v2/{org}/alerts/incidents/stats` - Get incident statistics - `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Trigger RCA analysis - `POST /api/{org}/agent/chat` - Generic agent chat endpoint - `POST /api/{org}/agent/chat/stream` - Streaming agent chat ## Database - **New tables**: `alert_incidents`, `alert_incident_alerts` - **Migration**: `m20251204_000001_create_alert_incidents_table` - **ORM**: SeaORM entities with full CRUD support ## UI Components - **IncidentList.vue**: Status filtering, pagination, statistics dashboard - **IncidentDetailDrawer.vue**: Full incident view with triggers/alerts/analysis - **SREChat.vue**: AI assistant panel with markdown rendering - Incidents tab in AlertList with unified view - AI sparkle icons for contextual chat access --- ## Enterprise Feature This is an **enterprise-only** feature with OSS stubs for compilation compatibility. - Non-enterprise builds: Handlers return `403 Forbidden` - Enterprise builds: Full functionality enabled ## Test Plan - [x] Build passes without `--features enterprise` - [x] Build passes with `--features enterprise` - [x] Create alerts that fire and verify incidents are created - [x] Verify incident list/detail APIs work correctly - [x] Verify status updates (open → acknowledged → resolved) - [x] Verify correlation logic (trace-based, service discovery, dimension-based) - [x] Verify RCA analysis triggers and displays correctly - [x] Verify SRE chat integration with alert/incident context --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>